Marquez Monthly Community Meeting
The Marquez Community Meeting occurs on the fourth Thursday of each month. Meetings are held on Zoom.
Next meeting: February 24, 2022
January 27, 2022
Attendees:
- Willy Lulciuc, Co-creator of Marquez
- Julien Le Dem, CTO of Datakin
- Ross Turk, VP of Marketing, Datakin
- Michael Collado, Staff Engineer, Datakin
- Minkyu Park, Senior Engineer, Datakin
- Peter Hicks, Senior Engineer, Datakin
- John Thomas, Support Engineer, Datakin
- Michael Robinson, Developer Relations Engineer, Datakin
- Kevin Mellott, Assistant Director of Data Engineering, Northwestern Mutual
Agenda:
- Marquez recent releases overview [Willy]
- Marquez release 0.21.0 overview
- Upgrade to Java17
- Marquez release 0.21.0 overview
- Migrating integrations to OpenLineage [Willy]
- Cloud-based development instance of Marquez via Gitpod [Peter]
- Open discussion
Meeting:
- Slides
- Zoom recording
- Passcode
Notes:
0.21.0 overview [Willy]
- Features:
- Bug fixes
- Removal of excess code
- Upgrade to Java17
- API image migrated
- Eclipse Temurin integrated
- All CI deployment updated to support Java17
- Discussion [Kevin, Willy, Michael C.]:
- Support for Java client possible in lower version
- Proposed: schedule separate meeting about this
Migrating integrations to OpenLineage [Willy]
- Spark library in Marquez now deprecated
- Going forward, use of OpenLineage Spark integration recommended
- review the docs about how to configure your instance
- remember to add underscore to marquez_airflow
- OpenLineage integration allows task listener
- workaround: import DAG from OpenLineage
- See the changelog: environment variables for the Airflow instance have changed
Cloud-based development instance of Marquez [Peter]
- Enabled by integration of Gitpod
- Docker image in the cloud with Marquez and UI
- Ideal for those not ready to install everything locally or who are having issues with their OS
- Fast (30 seconds), eliminates risk to codebase
- API also available
- Can be made private or public
- Big advantage: shareable within organizations via URL
- Supports everything one could do locally in VS Code or similar IDE
- Discussion [Willy, Peter, Kevin, Julien]:
- common use case: potential users want to see metadata from their org and share the tool
- potential side-effect: increase in Docker pulls
- availability of metrics unknown
- email address required
Open Discussion
- Advantages of possible move from CircleCI to Github Actions
- CircleCI downsides: outages, billing issues
- Proposed: moving to Github actions eventually after running both in parallel
- Kevin asked to experiment with Github Actions and report back
- Issue #1800
- Formal solution needed
- Willy proposed: deploy in two modes and use flags (Julien agreed)
- NodeID
- An easy win: add a field that returns a nodeID
- Willy proposed: prioritize in next release
Marquez Workflow Group Calendar Overview
Effective March 22, 2019: Group calendars are managed within LF AI Foundation Groups.io subgroups (mail lists); with each sub-group (mail list) having a unique group calendar. Meeting invites from these group calendars are sent to the applicable sub-group (mail list). In order to see the various group calendars you must:
Be logged into LF AI Foundation Groups.io
Be subscribed to the sub-group(mail-list) you're interested in
Thereafter, you will see all the calendars for the sub-groups you subscribe to under your LF AI Foundation Group Calendar via Groups.io OR
You can also view a specific group calendar via the Wiki (if the group has created a Wiki group calendar) whether you are a member of the sub-group (mail list) or not
View Instructions on How to Subscribe to LF AI Group Calendars
For detailed information on LF AI meeting management processes view this page: LF AI Foundation - Community Meetings and Calendars
Marquez Meetings List
Schedule | Title | Owner | Subgroup (mail list) | Purpose | Dial In Link |
|---|---|---|---|---|---|
| Day of Week (frequency) 00:00 AM/PM - 00:00 AM/PM (timezone) | Meeting Title (Zoom Account Used) | Meeting Owner/Moderator | marquez-mail-list@lists.lfai.foundation | Meeting Purpose | Zoom Name: https://zoom.us/... |
Marquez Group Calendar
- EDIT THE CALENDAR
Customise the different types of events you'd like to manage in this calendar.
#legIndex/#totalLegs - RESTRICT THE CALENDAR
Optionally, restrict who can view or add events to the team calendar.
#legIndex/#totalLegs - SHARE WITH YOUR TEAM
Grab the calendar's URL and email it to your team, or paste it on a page to embed the calendar.
#legIndex/#totalLegs - ADD AN EVENT
The calendar is ready to go! Click any day on the calendar to add an event or use the Add event button.
#legIndex/#totalLegs - SUBSCRIBE
Subscribe to calendars using your favourite calendar client.
#legIndex/#totalLegs