Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Airbyte v1 #77

Open
wants to merge 69 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
3fa0c97
change swagger ingress
Manoj-07 Jul 14, 2023
a6613cc
added pyathena_sql version
Manoj-07 Jul 19, 2023
12dced7
removed pyathena_sql version
Manoj-07 Jul 19, 2023
a1bd013
updated autoscalling for v2
Manoj-07 May 20, 2024
934ecdd
change in permsize
Manoj-07 Jul 2, 2024
67a0972
added blotout jar
Manoj-07 Jul 2, 2024
8166803
Create airbyte-v1 helm chart
astritzeqiri Dec 5, 2024
c4b4940
Set airbyte-v1 helm chart version
astritzeqiri Dec 6, 2024
fcf53c1
Update release.yaml
Manoj-07 Dec 6, 2024
ec83a31
Add default global.database.user on airbyte-v1
astritzeqiri Dec 6, 2024
3bf7560
Test removal of required vals on airbyte-v1
astritzeqiri Dec 6, 2024
23beafc
Test removal of required vals on airbyte-v1
astritzeqiri Dec 6, 2024
63c8e17
Add default global.database values
astritzeqiri Dec 6, 2024
4e851c5
Comment Chart.yaml dependencies on airbyte-v1
astritzeqiri Dec 6, 2024
f8fd59d
Comment Chart.yaml dependencies on airbyte-v1
astritzeqiri Dec 6, 2024
c7f18c6
Remove chart lock
astritzeqiri Dec 6, 2024
a4ad60a
Add chart loc
astritzeqiri Dec 6, 2024
eb380c7
Add the dependencies
astritzeqiri Dec 6, 2024
d7aa6f1
Read deps
astritzeqiri Dec 6, 2024
4988e86
Add local deps on airbyte-v1
astritzeqiri Dec 6, 2024
132388c
Change local deps on airbyte-v1
astritzeqiri Dec 6, 2024
50ca207
Readd deps
astritzeqiri Dec 6, 2024
16f3499
Fix gcs secrets yaml
astritzeqiri Dec 6, 2024
f6f4d87
Fix indent in gcs-log-creds-secret.yaml
mrexhepi Dec 6, 2024
9577f0a
Disable linting for release
mrexhepi Dec 6, 2024
701711c
trigger release action
mrexhepi Dec 6, 2024
63bcfac
Disable kubeval
mrexhepi Dec 6, 2024
1ae0de3
Pin charts airbyte-v1
mrexhepi Dec 6, 2024
018475d
Pin charts airbyte-v1
mrexhepi Dec 6, 2024
34361ff
Fix secrets and cp
mrexhepi Dec 6, 2024
82bdd0d
Update appVersion to 1.2.0
mrexhepi Dec 6, 2024
5f4d695
Create new workflow only for airbyte-v1
astritzeqiri Dec 10, 2024
eafa77e
Uncomment kubeval
astritzeqiri Dec 10, 2024
89398b1
Yaml lint
astritzeqiri Dec 10, 2024
4c7ad44
fix paths
astritzeqiri Dec 10, 2024
49c5547
fix paths on ci.yml
astritzeqiri Dec 10, 2024
a772335
Ignore airbyte-v1 on kubeval
astritzeqiri Dec 10, 2024
046d48b
Reorganize worflows
astritzeqiri Dec 10, 2024
9ef102d
Fix release workflow
astritzeqiri Dec 10, 2024
409f822
Fix kubeval ignore of airbyte-v1, delete airbyte-v1 workflows
astritzeqiri Dec 10, 2024
befc51a
Ignore all airbyte-v1 chart.yml from kubeval
astritzeqiri Dec 10, 2024
d8efe8a
Add imagepullsecrets value secret
mrexhepi Dec 19, 2024
8d9d527
update airbyte-v1 chart to v1.1.1
mrexhepi Dec 19, 2024
ab7b042
test release v1
mrexhepi Dec 19, 2024
554adfb
update airbyte-v1 chart to v1.0.1
mrexhepi Dec 19, 2024
f9fe22f
Enable pull docker image from dockerhub
mrexhepi Dec 20, 2024
2f0911e
Add rolling update deployment for airbyte-server
mrexhepi Dec 23, 2024
46e336f
Update airbyte-1 helm chart
mrexhepi Dec 24, 2024
4cb3a5f
test airbyte-v1 1.3.0 version
mrexhepi Dec 30, 2024
b334dc2
update kube secret
mrexhepi Dec 30, 2024
b896d22
update helm chart airbyte v1.3.1
mrexhepi Dec 30, 2024
88bacdc
update helm chart airbyte v1.3.2
mrexhepi Dec 30, 2024
ffa25b6
update helm chart airbyte v1.3.2
mrexhepi Dec 30, 2024
f53fcdd
Add blotout vars to configmap
mrexhepi Jan 8, 2025
fc807c0
Add vars for aws keys
mrexhepi Jan 8, 2025
8cccebc
Add more vars to cmp of airbyte-v1
mrexhepi Jan 8, 2025
548b8ac
Cleanup unused airbyte-v1 dir
mrexhepi Jan 8, 2025
85770c2
Add blotout env to server template
mrexhepi Jan 9, 2025
4a2f875
update airbyte-v1 chart to v1.3.5
mrexhepi Jan 9, 2025
dedbb16
test local charts,v1.3.6
mrexhepi Jan 9, 2025
eb8330d
Test server env vars, v1.3.7
mrexhepi Jan 9, 2025
6473414
Fix BLOTOUT_AUTH_ENDPOINT and BLOTOUT_BASE_URL vars
mrexhepi Jan 9, 2025
ea2cb9a
Fix BLOTOUT_AUTH_ENDPOINT and BLOTOUT_BASE_URL vars
mrexhepi Jan 9, 2025
8a0404f
Set resource requests and limitation for critical airbyte-v1 components
mrexhepi Jan 17, 2025
a81b76e
test remove not needed vars
mrexhepi Jan 17, 2025
ff86e0a
Set resource requests and limitation for critical airbyte-v1 components
mrexhepi Jan 17, 2025
d59c026
Increase chart version airbyte-v1 to v1.3.9
mrexhepi Jan 21, 2025
4102e3f
upgrade airbyte-v1 helm release to v1.3.10
mrexhepi Jan 31, 2025
d1d662d
change in airbyte resources
Manoj-07 Feb 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/kubeval.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash
set -euo pipefail

CHART_DIRS="$(git diff --find-renames --name-only "$(git rev-parse --abbrev-ref HEAD)" remotes/origin/main -- helm | grep '[cC]hart.yaml' | sed -e 's#/[Cc]hart.yaml##g')"
CHART_DIRS="$(git diff --find-renames --name-only "$(git rev-parse --abbrev-ref HEAD)" remotes/origin/main -- helm | grep '[cC]hart.yaml' | grep -v 'helm/airbyte-v1' | sed -e 's#/[Cc]hart.yaml##g')"
KUBEVAL_VERSION="0.14.0"
SCHEMA_LOCATION="https://raw.githubusercontent.com/instrumenta/kubernetes-json-schema/master/"

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ name: Lint and Test Charts

on:
pull_request:
paths:
- 'helm/**'
paths-ignore:
- 'helm/airbyte-v1/**'

jobs:
lint-chart:
Expand Down Expand Up @@ -56,4 +56,4 @@ jobs:
uses: helm/chart-testing-action@main
with:
command: install
config: .github/ct.yaml
config: .github/ct.yaml
7 changes: 5 additions & 2 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ on:
branches:
- main
- eks_upgrade
paths:
- 'helm/**'
- airbyte_v1
# paths-ignore:
# - 'helm/airbyte-v1/**'
release:
types: [published, created]

Expand All @@ -20,3 +21,5 @@ jobs:
with:
token: ${{ secrets.BO_CHARTS_HELM_TOKEN }}
charts_dir: helm
# Due to airbyte-v1 version incompatility with the current config of gh actions it always fails on linting so we switched it off
linting: off
26 changes: 26 additions & 0 deletions helm/airbyte-v1/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/

ci.sh
.cache/
51 changes: 51 additions & 0 deletions helm/airbyte-v1/Chart.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
version: 1.17.1
- name: airbyte-bootloader
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: temporal
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: temporal-ui
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: webapp
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: server
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: worker
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: workload-api-server
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: workload-launcher
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: pod-sweeper
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: metrics
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: cron
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: connector-builder-server
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: connector-rollout-worker
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: keycloak
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- name: keycloak-setup
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
digest: sha256:f0a8ad517d58f33d56b16654e9b868a981e7dbac7af214473a8f486dc5a2e563
generated: "2024-12-09T21:59:21.985259637Z"
72 changes: 72 additions & 0 deletions helm/airbyte-v1/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
apiVersion: v2
appVersion: 1.3.0
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
tags:
- bitnami-common
version: 1.x.x
- condition: airbyte-bootloader.enabled
name: airbyte-bootloader
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: temporal.enabled
name: temporal
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: temporal-ui.enabled
name: temporal-ui
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: webapp.enabled
name: webapp
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: server.enabled
name: server
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: worker.enabled
name: worker
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: workload-api-server.enabled
name: workload-api-server
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: workload-launcher.enabled
name: workload-launcher
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: pod-sweeper.enabled
name: pod-sweeper
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: metrics.enabled
name: metrics
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: cron.enabled
name: cron
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: connector-builder-server.enabled
name: connector-builder-server
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: connector-rollout-worker.enabled
name: connector-rollout-worker
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: keycloak.enabled
name: keycloak
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
- condition: keycloak-setup.enabled
name: keycloak-setup
repository: https://airbytehq.github.io/helm-charts/
version: 1.3.0
description: Helm chart to deploy airbyte
name: airbyte-v1
type: application
version: "v1.3.11"
2 changes: 2 additions & 0 deletions helm/airbyte-v1/charts/airbyte-bootloader/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Charts are downloaded at install time with `helm dep build`.
charts
6 changes: 6 additions & 0 deletions helm/airbyte-v1/charts/airbyte-bootloader/Chart.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
version: 1.17.1
digest: sha256:dacc73770a5640c011e067ff8840ddf89631fc19016c8d0a9e5ea160e7da8690
generated: "2022-10-17T18:35:11.238826963Z"
12 changes: 12 additions & 0 deletions helm/airbyte-v1/charts/airbyte-bootloader/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: v2
appVersion: 1.3.0
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
tags:
- bitnami-common
version: 1.x.x
description: Helm chart to deploy airbyte-bootloader
name: airbyte-bootloader
type: application
version: 1.3.0
47 changes: 47 additions & 0 deletions helm/airbyte-v1/charts/airbyte-bootloader/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# airbyte-bootloader

![Version: 0.67.17](https://img.shields.io/badge/Version-0.67.17-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: dev](https://img.shields.io/badge/AppVersion-dev-informational?style=flat-square)

Helm chart to deploy airbyte-bootloader

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| https://charts.bitnami.com/bitnami | common | 1.x.x |

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| containerSecurityContext | object | `{}` | |
| enabled | bool | `true` | |
| env_vars | object | `{}` | |
| extraContainers | list | `[]` | |
| extraEnv | list | `[]` | |
| extraInitContainers | list | `[]` | |
| extraLabels | object | `{}` | |
| extraSelectorLabels | object | `{}` | |
| extraVolumeMounts | list | `[]` | |
| extraVolumes | list | `[]` | |
| global.database.secretName | string | `""` | |
| global.database.secretValue | string | `""` | |
| global.deploymentMode | string | `"oss"` | |
| global.env_vars | object | `{}` | |
| global.extraContainers | list | `[]` | |
| global.extraLabels | object | `{}` | |
| global.extraSelectorLabels | object | `{}` | |
| global.secretName | string | `""` | |
| global.secrets | object | `{}` | |
| global.serviceAccountName | string | `"placeholderServiceAccount"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"airbyte/bootloader"` | |
| nodeSelector | object | `{}` | |
| podAnnotations | object | `{}` | |
| podLabels | object | `{}` | |
| resources.limits | object | `{}` | |
| resources.requests | object | `{}` | |
| secrets | object | `{}` | |
| tolerations | list | `[]` | |

51 changes: 51 additions & 0 deletions helm/airbyte-v1/charts/airbyte-bootloader/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "airbyte.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "airbyte.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "airbyte.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "airbyte.labels" -}}
helm.sh/chart: {{ include "airbyte.chart" . }}
{{ include "airbyte.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "airbyte.selectorLabels" -}}
app.kubernetes.io/name: {{ include "airbyte.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
17 changes: 17 additions & 0 deletions helm/airbyte-v1/charts/airbyte-bootloader/templates/_images.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- define "imageUrl" -}}
{{- $img := index . 0 -}}
{{- $root := index . 1 -}}

{{- $reg := $root.Values.global.image.registry -}}
{{/* ensure the registry has a trailing slash, if set */}}
{{- $reg = (ternary $reg (printf "%s/" (trimSuffix "/" $reg)) (empty $reg)) -}}

{{/* some images are defined as a string instead of an object (busybox, curl, connector sidecar, etc) */}}
{{- if (eq (typeOf $img) "string") -}}
{{- printf "%s%s" $reg (tpl $img $root) | quote -}}
{{- else -}}
{{- $tag := coalesce $img.tag $root.Values.global.image.tag $root.Chart.AppVersion -}}
{{- printf "%s%s:%s" $reg $img.repository $tag | quote -}}
{{- end -}}

{{- end -}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Create secrets only for the local deployment
{{- if .Values.secrets }}
apiVersion: v1
kind: Secret
metadata:
name: bootloader-secrets
labels:
{{- include "airbyte.labels" . | nindent 4 }}
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-weight: "-1"

type: Opaque
data:
{{- range $k, $v := mergeOverwrite .Values.secrets .Values.global.secrets }}
{{ $k }}: {{ if $v }}{{ $v | b64enc }} {{else}}""{{end}}
{{- end }}
{{- end }}
Loading
Loading