Bump version to 4.10.1 #430
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
f-galland
approved these changes
Sep 24, 2024
AlexRuiz7
added a commit
that referenced
this pull request
Oct 18, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Remove packaging_scripts folder * Remove duplicated files * Fix build.yml --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Oct 22, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Remove packaging_scripts folder * Remove duplicated files * Fix build.yml --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Nov 5, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Merge 4.10.1 into 4.10.2 (#513) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.10.0 into 4.10.1 (#511) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 * Fix release date for 4.10.0 in RPM spec file (#471) * Preserve status of wazuh-indexer on upgrade (#498) * Update pre and post inst scripts for deb and rpm to store and restore service status * Update prerm script to avoid stopping the service on upgrade * Remove extra spaces and update rpm restart command * Merge 4.9.2 into 4.10.0 (#510) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 * Support new version 4.9.2 (#494) * Support new version 4.9.2 * Add estimated release date for 4.9.2 * Fix estimates release date for 4.9.2 * Fix 4.9.1 release notes title --------- Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Nov 8, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Remove packaging_scripts folder * Remove duplicated files * Fix build.yml --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Nov 8, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Merge 4.10.1 into 4.10.2 (#513) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.10.0 into 4.10.1 (#511) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 * Fix release date for 4.10.0 in RPM spec file (#471) * Preserve status of wazuh-indexer on upgrade (#498) * Update pre and post inst scripts for deb and rpm to store and restore service status * Update prerm script to avoid stopping the service on upgrade * Remove extra spaces and update rpm restart command * Merge 4.9.2 into 4.10.0 (#510) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 * Support new version 4.9.2 (#494) * Support new version 4.9.2 * Add estimated release date for 4.9.2 * Fix estimates release date for 4.9.2 * Fix 4.9.1 release notes title --------- Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Nov 12, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Create branch 5.0.0 (#154) * Create branch 5.0.0 * Fix CHANGELOG.md * Update `build` workflow to build indexer plugins (#360) * Update build workflow to include Wazuh plugins * Try new approach to build wazuh-indexer with plugins * Remove old code * Remove ADMINS.md artifacts benchmarks build build.gradle buildSrc CHANGELOG.md client codecov.yml CODE_OF_CONDUCT.md CONTRIBUTING.md DEVELOPER_GUIDE.md dev-tools distribution docker docs doc-tools ecs gradle gradle.properties gradlew gradlew.bat integrations libs licenses LICENSE.txt MAINTAINERS.md modules NOTICE.txt packaging_scripts plugins protobuf-java-NOTICE.txt qa README.md release-notes RELEASING.md rest-api-spec sandbox SECURITY.md server settings.gradle test TESTING.md Vagrantfile VERSION whitesource.config step * Sync mavel local path across jobs * Fix versioning of wazuh-indexer-plugins * Fix versioning of wazuh-indexer-plugins * Pass version and revision to publishToMavenLocal * Add version check test * Format files * Use upload-artifact and download-artifact to share the plugins' zips between jobs * Add repo path * Fix plugin name * Roll back * Remove exit 1 * Fix relative path to the plugins * List plugins folder * Fix relative path * again * Change relative path to absolute * Clean code * Update README.ms * Apply naming convention * Add breif steps to build wazuh-indexer with plugins * Skip job to build plugins on no input * Improve conditional * Remove build-plugins job from build's job dependencies * Roll back * Add tooling to generate the agents index template (#370) * Merge 4.10.0 into master (#379) * Merge 4.9.2 into 4.10.0 (#378) * Update changelog * Remove `alerts.json` references and manager integrations (#385) * Remove references to alerts.json and filebeat off events generator * Remove compose files and logstash pipelines * Remove ossec references from sample events * Remove old compose files for integrations --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix build.gradle (#381) (#384) * Fix build.gradle * Fix build.gradle * Undo changes * Add issue template for Indexer-Dashboard packages testing (#393) * Add stateless index template definition (#395) * Add stateless index template definition Event generator is pending * Update to 8.11.0 * Update ECS generator * Remove event generator for stateless ECS module * Remove commented code * Fix typo * Add states-inventory-packages index template definition (#399) * Add stateless index template definition Event generator is pending * Update to 8.11.0 * Adding template mappings and settings for states-inventory-packages index * Fix indentation issue in subset.yml * Remove event generators * Remove duplicated code con ECS generator * Add custom fields for states-inventory-packages * Remove hidden flag on index template --------- Co-authored-by: Álex Ruiz <[email protected]> * Add states-inventory-processes index template definition (#401) * Add stateless index template definition Event generator is pending * Update to 8.11.0 * Adding template mappings and settings for states-inventory-processes index * Fix indentation issue in subset.yml * Add process.tty as a custom field * Update states-inventory-processes index template definition * Remove events generators * Remove duplicated code * Remove hidden flag on index template --------- Co-authored-by: Álex Ruiz <[email protected]> * Add states-inventory-system index template definition (#403) * Add stateless index template definition Event generator is pending * Update to 8.11.0 * Adding template mappings and settings for states-inventory-system index * Remove hidden flag, correct subset.yml indentation * Fix stuff --------- Co-authored-by: Álex Ruiz <[email protected]> * Add states-vulnerabilities index template definition (#405) * Add stateless index template definition Event generator is pending * Update to 8.11.0 * Adding template mappings and settings for states-inventory-vulnerabilities index * Remove event generator script * Remove hidden flag * Fix subset.yml indentation * Recycle ecs/vulnerability-detector * Add yaml header --------- Co-authored-by: Álex Ruiz <[email protected]> * Add states-fim index template definition (#397) * Add stateless index template definition Event generator is pending * Update to 8.11.0 * Adding ecs mapping files for FIM index * Fix indentation issue in subset.yml * Remove hidden flag and event_generator * Rename states-inventory-fim folder * Fix subset.yml names --------- Co-authored-by: Álex Ruiz <[email protected]> * Include Command Manager plugin to the build workflow (#408) * Include Command Manager plugin to the build workflow * Remove 'github.event.' * Remove double slash * Update artifact path * Add commands index template definition (#413) * Add commands index template definition * Change oreder_id data type * Build & Assemble reporting plugin (#431) * Build & Assemble reporting plugin * Add working-directto ls * Swap reporting plugin in wazuh-indexer package (specs) * Fix changelog chronological order * Normalize artifact names * Use env.plugin_name * Add events generator for the Commands Manager plugin (#433) The event generator can gencreate and push sample events to the Command Manager API or to the Indexer API * Update commands index defition (#437) Change ID types to keywords * Update commands index data model (#453) * Update commands index data model * Update commands event generator * Move agent fields as extended * Merge 4.10.2 into master (#475) * Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc35007c0fbd8c6f0a54d35e9118a1936fd08f1. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability ma…
AlexRuiz7
added a commit
that referenced
this pull request
Nov 18, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Remove packaging_scripts folder * Remove duplicated files * Fix build.yml --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Nov 18, 2024
* Init wazuh-indexer (#3) * Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Create codeql.yml Signed-off-by: Álex Ruiz <[email protected]> * Update dependabot.yml Signed-off-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#30) Signed-off-by: Álex Ruiz <[email protected]> * Add ECS mappings generator (#36) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]> * Add default query fields to vulnerability detector index (#40) * Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]> * Create gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Update gradle_build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add a script to configure the rollover policy (#49) * Update ISM init script (#50) * Fix bug with -i option (#51) * Fix bug with -i option * Improve error handling * Update min_doc_count value (#52) * Improve ISM init script (#57) * Improve ISM init script * Change log file path * Update distribution files (#59) * Update config files * Add VERSION file * Update documentation of the ECS tooling (#67) * Add workflow for package generation (#65) * Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code * Add docker compose environment (#66) * Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename packages to wazuh-indexer (#69) * Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck * Update vulnerability index mappings (#75) * Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector * Update `indexer-ism-init.sh` (#81) Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to assemble packages (#85) * Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act * Fix yellow cluster state (#95) * Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line * Update ism-init script (#97) * Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]> * Add tools to assemble DEB packages (#96) * Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script * Update README.md Signed-off-by: Álex Ruiz <[email protected]> * Build scripts and GH workflows artifacts naming fix (#112) * Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]> * Use short SHA as Git reference in packages naming (#100) * Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Update issue templates (#127) * Fix RPM package references to /var/run (#119) * Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing post-install message from wazuh-indexer.rpm.spec (#131) * Add tests to the packages building process (#132) Runs the workflow on pull request changes * Get Wazuh version from VERSION file (#122) * Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Removing /usr/share/lintian/overrides/wazuh-indexer from deb packages (#130) Co-authored-by: Álex Ruiz <[email protected]> * Add `wazuh-template.json` to packages (#116) * Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]> * Adding Debian packaging config files from Opensearch (#118) * Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]> * Fix Build workflow to run on push events (#134) * Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator * Use maven for plugin download (#139) * Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]> * Add new custom field to the vulnerability detector index (#141) * Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings * Fine tuning permissions on assembled packages (#137) * Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]> * Init. Amazon Security Lake integration (#143) * Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]> * Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> * Update Gradle setup action (#182) * Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability-states fields (#177) * Update vulnerability-states fields Adds wazuh.schema.version * Update events generator * Automate package's testing (#178) * Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]> * Remove ecs.version from query.default_fields (#184) * Upload packages to S3 (#179) * Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]> * Add bash to Docker dev image (#185) * Update wazuh-states-vulnerabilities index mapping (#191) * Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields * Add pipeline to generate release packages (#193) * Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]> * Build Docker images (#194) * Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]> * Add on.workflow_call to build_single.yml workflow (#200) Allows invocation usin the GH API * Add Pyhton module to implement Amazon Security Lake integration (#186) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Replace choice with string on workflow_call (#207) * Use AWS_REGION secret (#209) * Add Lambda function for the Amazon Security Lake integration (#189) * Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]> * Bump Java version in Docker environments (#210) * Fix access denied error during log rotation (#212) * Save intermediate OCSF files to an S3 bucket (#218) * Fix Parquet files format (#217) * Fix mapping to Detection Finding OCSF class (#220) * Map events to OCSF's Security Finding class (#221) * Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error * Add ID input to workflows (#229) * Added id input * Changed name to run-name * Add OPENSEARCH_TMPDIR variable to service and create directory in packages accordingly (#231) * Improve workflow's run-name with tagret system and architeture (#237) * Add documentation for the Amazon Security Lake integration (#226) * Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]> * Rename environment variable (#240) * Remove maintainer-approval.yml (#241) * Improve logging and error handling on ASL Lambda function (#242) * Update .gitattributes (#243) * Change . for : in debian's postinst (#245) * Add integration with Elastic (#248) * Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]> * Added S3 URI output to package generation upload (#249) * Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]> * Add OpenSearch integration (#258) * Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]> * Add Splunk integration (#257) * Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add Manager to Elastic integration (#266) * Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]> * Add Manager to Splunk integration (#268) * Add Manager to OpenSearch integration (#267) * Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README * Attempt nr.2 to fix #277 (#280) * Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata * Remove references to indexer-ism-init.sh and wazuh-template.json (#281) * Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]> * Bump 4.10.0 (#272) * Merge 4.9.1 into 4.10.0 (#358) * Merge 4.9.1 into 4.10.0 (#358) --------- Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.9.2 into 4.10.0 (#378) * Fix build.gradle (#381) * Fix build.gradle * Fix build.gradle * Undo changes * Remove old compose files for integrations (#386) * Delete integrations/docker/amazon-security-lake.yml Signed-off-by: Álex Ruiz <[email protected]> * Delete integrations/docker/config directory Signed-off-by: Álex Ruiz <[email protected]> * Update vulnerability detector index template (#383) * Update VD index template * Remove host.os.family * Merge 4.9.1 into 4.10.0 (#426) * Fix Performance Analyzer service file (#391) * Update SECURITY.md (#411) * Remove prompt about configuration file overwrites on package upgrade (#410) * Make new config files install with .new prefix * Fix errors and add .new prefix to /etc/init.d/wazuh-indexer * Fix errors in build.sh and assemble.sh * Revert "Fix errors in build.sh and assemble.sh" This reverts commit 5dc3500. * Using noreplace on config files for rpm * Fix issues in debmake.sh * Revert changes to Debian packages --------- Co-authored-by: Álex Ruiz <[email protected]> * Update SECURITY.md (#415) Signed-off-by: Raul Del Pozo Moreno <[email protected]> * Add Release Notes 4.9.1-rc1 (#421) --------- Signed-off-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> * Bump version to 4.10.1 (#430) * Support new version 4.10.2 (#441) * Enable assembly of ARM packages (#444) * Merge 4.10.1 into 4.10.2 (#473) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file * Fix release date for 4.10.0 in RPM spec file --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Merge 4.10.1 into 4.10.2 (#513) * Merge 4.10.0 into 4.10.1 (#470) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> * Fix release date for 4.10.0 in RPM spec file Signed-off-by: Álex Ruiz <[email protected]> * Merge 4.10.0 into 4.10.1 (#511) * Upgrade integrations to the last version (#447) * Upgrade third-party integrations to latest product versions (#368) * Upgrade third-party integrations to latest product versions * Improve comtability matrix * Change versions in /integrations/.env Signed-off-by: Malena Casas <[email protected]> * Fix Splunk integrations (#362) * Add table with the version of the integrations * Update CHANGELOG.md Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> * Merge 4.9.1 into 4.10.0 (#454) * Prepare 4.9.1-rc2 (#436) * Update docker/README.md (#438) * Support new stage 4.9.1-rc3 (#443) * Update operational--integrations_maintenance_request.md (#449) Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> * Fix Github Actions build process dependency errors (#457) * Switch from latest to 22.04 runner * Remove non-existant packages from workflow provisioner * Remove freeglut3 from provision.sh * Update calendarTime and scan_date fields type (#458) * Merge 4.9.1 into 4.10.0 (#469) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 * Fix release date for 4.10.0 in RPM spec file (#471) * Preserve status of wazuh-indexer on upgrade (#498) * Update pre and post inst scripts for deb and rpm to store and restore service status * Update prerm script to avoid stopping the service on upgrade * Remove extra spaces and update rpm restart command * Merge 4.9.2 into 4.10.0 (#510) * Support for v4.9.1-alpha4 (#461) * Prepare final release notes for 4.9.1 * Support new version 4.9.2 (#494) * Support new version 4.9.2 * Add estimated release date for 4.9.2 * Fix estimates release date for 4.9.2 * Fix 4.9.1 release notes title --------- Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Malena Casas <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Fede Galland <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Raul Del Pozo Moreno <[email protected]> Signed-off-by: Malena Casas <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Raul Del Pozo Moreno <[email protected]> Co-authored-by: Malena Casas <[email protected]> Co-authored-by: JuanGarriuz <[email protected]> Co-authored-by: Kevin Ledesma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR initializes the 4.10.1 branch.
Issues Resolved
Closes #429
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.