Marquez Monthly Community Meeting

The Marquez Community Meeting occurs on the fourth Thursday of each month. Meetings are held on Zoom.

February 24, 2022

Attendees:

TSC:

• Willy Lulciuc, Co-creator of Marquez
• Michael Collado, Staff Engineer, Datakin

And:

• Minkyu Park, Senior Engineer, Datakin
• Michael Robinson, Developer Relations Engineer, Datakin
• Ross Turk, VP of Marketing, Datakin

Agenda:

• Review of integrations to create runs and associate metadata with runs (replaced with OpenLineage)
• Demo: How to collect OpenLineage events with the lineage API to send metadata to Marquez
• Dataset lifecycle management

Meeting:

• Slides
• Zoom recording
• Passcode: Q89sjr.c

January 27, 2022

Attendees:

TSC:

• Willy Lulciuc, Co-creator of Marquez
• Julien Le Dem, CTO of Datakin
• Michael Collado, Staff Engineer, Datakin
• Peter Hicks, Senior Engineer, Datakin
• Kevin Mellott, Assistant Director of Data Engineering, Northwestern Mutual

And:

• Ross Turk, VP of Marketing, Datakin
• Minkyu Park, Senior Engineer, Datakin
• John Thomas, Support Engineer, Datakin
• Michael Robinson, Developer Relations Engineer, Datakin

Agenda:

• Marquez recent releases overview [Willy] 
  • Marquez release 0.21.0 overview
    • Upgrade to Java17
• Migrating integrations to OpenLineage [Willy]
• Cloud-based development instance of Marquez via Gitpod [Peter]
• Open discussion

Meeting:

• Slides
• Zoom recording
• Passcode: Ef2^5Wwg

Notes:

0.21.0 overview [Willy]

• Features:
  • Bug fixes
  • Removal of excess code
  • Upgrade to Java17
    • API image migrated
    • Eclipse Temurin integrated
    • All CI deployment updated to support Java17
• Discussion [Kevin, Willy, Michael C.]:
  • Support for Java client possible in lower version
  • Proposed: schedule separate meeting about this

Migrating integrations to OpenLineage [Willy]

• Spark library in Marquez now deprecated
• Use of OpenLineage Spark integration recommended going forward
  • review the docs about how to configure your instance
  • remember to add underscore to marquez_airflow
• OpenLineage integration allows task listener
  • workaround: import DAG from OpenLineage
• See the changelog: environment variables for the Airflow instance have changed

Cloud-based development instance of Marquez [Peter]

• Enabled by integration of Gitpod
• Docker image in the cloud with Marquez and UI
• Ideal for those not ready to install everything locally or who are having issues with their OS
• Fast (30 seconds), eliminates risk
• API also available
• Can be made private or public
• Big advantage: shareable within organizations via URL
• Supports everything one could do locally in VS Code or similar IDE
• Discussion [Willy, Peter, Kevin, Julien]:
  • common use case: potential users want to see metadata from their org and share the tool
  • potential side-effect: increase in Docker pulls
  • availability of metrics unknown
  • email address required

Open Discussion

• Advantages of possible move from CircleCI to Github Actions 
  • CircleCI downsides: outages, billing issues [Willy]
  • Julien proposed: moving to Github actions eventually after running both in parallel
  • Kevin asked to experiment with Github Actions and report back
• Issue #1800: add support for table operations reported from OpenLineage
  • Formal solution needed [Willy]
  • Willy proposed: deploy in two modes and use flags (Julien agreed)
• NodeID
  • An easy win: add a field that returns a nodeID [Willy]
  • Willy proposed: prioritize in next release

Marquez Workflow Group Calendar Overview

Effective March 22, 2019: Group calendars are managed within LF AI Foundation Groups.io subgroups (mail lists); with each sub-group (mail list) having a unique group calendar. Meeting invites from these group calendars are sent to the applicable sub-group (mail list). In order to see the various group calendars you must:

• Be logged into LF AI Foundation Groups.io

• Be subscribed to the sub-group(mail-list) you're interested in

• Thereafter, you will see all the calendars for the sub-groups you subscribe to under your LF AI Foundation Group Calendar via Groups.io OR

• You can also view a specific group calendar via the Wiki (if the group has created a Wiki group calendar) whether you are a member of the sub-group (mail list) or not

  • Example: LF AI TAC Group Calendar (tac-general@lists...) via Wiki

View Instructions on How to Subscribe to LF AI Group Calendars

For detailed information on LF AI meeting management processes view this page: LF AI Foundation - Community Meetings and Calendars

Marquez Meetings List

Marquez Group Calendar