Exception thrown in prometheus exporter when trying to scrape about airflow-exporter

[elephantum]

@rmn36 can you please check with the latest release if everything works for you?

[elephantum]

Can you elabprate on your setup? Which version of Airflow do you use?

[rmn36]

Using version 1.9.0-4 based on this popular docker image with Python 3: https://github.com/puckel/docker-airflow

Issue appears to be that the Sample object has more than just name, label and value now. this may be a change on the prometheus client side.

[2018-10-11 21:30:34 +0000] [35] [INFO] Worker exiting (pid: 35)
Sample(name='process_virtual_memory_bytes', labels={}, value=275132416.0, timestamp=None, exemplar=None)

Hotfix is straightforward. Something like this:

for name, labels, value, timestamp, exemplar in metric.samples:

but that does not make sure of the additional information in any way.

Edit: this appears to fix the issue in Docker but when running in Kubernetes it still fails but with a different exception. Still investigating

[rmn36]

It now fails on one of the SQL queries.

This function:

def get_task_state_info():
    '''get task info
    :return task_info
    '''
    task_status_query = Session.query(
        TaskInstance.dag_id, TaskInstance.task_id,
        TaskInstance.state, func.count(TaskInstance.dag_id).label('value')
    ).group_by(TaskInstance.dag_id, TaskInstance.task_id, TaskInstance.state).subquery()
    return Session.query(
        task_status_query.c.dag_id, task_status_query.c.task_id,
        task_status_query.c.state, task_status_query.c.value, DagModel.owners
    ).join(DagModel, DagModel.dag_id == task_status_query.c.dag_id).all()

creates this query:

SELECT anon_1.dag_id AS anon_1_dag_id, 
       anon_1.task_id AS anon_1_task_id, 
       anon_1.state AS anon_1_state, 
       anon_1.value AS anon_1_value, 
       dag.owners AS dag_owners 
FROM (SELECT task_instance.dag_id AS dag_id, 
             task_instance.task_id AS task_id, 
             task_instance.state AS state, 
             count(task_instance.dag_id) AS value 
      FROM task_instance 
      GROUP BY task_instance.dag_id, task_instance.task_id, task_instance.state ) AS anon_1 
JOIN dag ON dag.dag_id = anon_1.dag_id

causes this error:
ERROR: relation "task_instance" does not exist at character 312

This seems to refers to the FROM task_instance in the subquery.

This is the Dockerfile I'm using the create the image with the plugin and client

FROM puckel/docker-airflow:1.9.0-4

USER root

ADD ./plugins /usr/local/airflow/plugins
RUN pip3 install prometheus_client

USER airflow

[elephantum]

Thanks, we’ll try to reproduce your conditions and fix the problem.

[rmn36]

@elephantum Thanks! It appears that when the plugin and client are installed the postgresql database isn't initialized properly causing the task_instance table to not be present. The only difference in the postgresql logs I can find is that when the plugin and client are installed the postgresql process complains about an "incomplete startup packet"

LOG: incomplete startup packet

EDIT It might actually be a timing issue that the code is trying to access postgres before the database is actually created and it's causing this error and crashing the airflow process.

EDIT 2: everything I said was wrong. The database gets created fine. I think part of it might be a timing issue. However, the failure seems to be happening when loading dags

[rmn36]

@elephantum fixed by upgrading psycopg2

pip3 install psycopg2 -U