![]() We will directly access this internal database to see what has been stored, with a focus on the following six tables: If you have yet to use Airbyte and would like to follow along, continue with the next two chapters.Īirbyte logs its status in an internal Postgres database. If so, please jump to step 3, where we create the needed views and dashboard. You probably use Airbyte already, and thus the interest in monitoring. Or follow the event-based approachmentioned exposing these metrics. You could also copy the data to BigQuery/Snowflake and run dbt on that data. But it could be if you replicate Airbyte tables to a read-only schema or database. ![]() □This tutorial is not meant to use in production as we query the internal Postgres database of Airbyte. We use Metabase, but I also tried Superset and Rill Data. ![]() □ All Metrics are implemented with dbt views therefore, you can put any BI toolon top. Feel free to fork and play around with it and create issues or PRs for improvements □. ℹ️ All code shown here is open on GitHub in the Open Data Stack Repo. Our dashboard approach is interesting, as you can see everything at one glance, add filters and drill-downs, or correlate with other data you might have. Today you have several options to either monitor via Airbyte UI by clicking through the syncs and logs or set up a more sophisticated systemby channeling metrics events through Prometheus. It will essentially allow you to get an operational view of your current running syncs and a high-level overview of what happened in the past. We will implement an Airbyte monitoring dashboard with dbt and Metabase on a locally deployed Airbyte instance. ![]() This tutorial is the first part of the Series“Building an Open Data Stack”, where we explore different use cases with open-source tools and open standards. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |