Start = DummyOperator(task_id='run_this_first', dag=dag) 'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10)) View the logs for the individual pods to know when they're up ( kubectl logs -f ) Load the sample airflow DAGĬopy the sample airflow dag from into a file named k8s-sample.py from airflow import DAGįrom _pod_operator import KubernetesPodOperatorįrom _operator import DummyOperator Note: The various airflow containers will take a few minutes until their fully operable, even if the kubectl status is RUNNING. Helm install -namespace "airflow" -name "airflow" -f airflow.yaml ~/src/charts/incubator/airflow/ This file sets the DB connection string ( sql_alchemy_conn = This is configurable via an enviroment variable in airflow's current master branch, but not in the 1.10 release. Helm dependency build ~/src/charts/incubator/airflow/Ĭopy the configmap-airflow-worker.yaml file attached below to. Helm installed and initialized in the minikube instanceĮdit the Dockerfile and add a line in the RUN command to install the kubernetes python packageĬonfigure docker to execute within the minikube VMīuild the and tag the image within the minikube VMĭocker build -t airflow-docker-local:1 Install the helm airflow chartĬreate a airflow.yaml helm config file: airflow:ĪIRFLOW_KUBERNETES_WORKER_CONTAINER_REPOSITORY: airflow-docker-localĪIRFLOW_KUBERNETES_WORKER_CONTAINER_TAG: 1ĪIRFLOW_KUBERNETES_WORKER_CONTAINER_IMAGE_PULL_POLICY: NeverĪIRFLOW_KUBERNETES_WORKER_SERVICE_ACCOUNT_NAME: airflowĪIRFLOW_KUBERNETES_DAGS_VOLUME_CLAIM: airflowįetch the helm dependencies for the airflow chart.This guide works with the airflow 1.10 release, however will likely break or have unnecessary extra steps in future releases (based on recent changes to the k8s related files in the airflow source). The steps below bootstrap an instance of airflow, configured to use the kubernetes airflow executor, working within a minikube cluster.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |