Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview

During the march 11th GB F2F meeting, the board The LF AI Governing Board recommended having a reference ML workflow / stack, and showcasing where current LF AI hosted projects fall. The idea is to have a reference architecture and to allow us to identify gaps in our project portfolio so we can focus on project recruitment in areas that we need most, leading us to provide a reference implementation using LF AI hosted projects.

Mail

Mailing List

Please subscribe to the TAC ML Workflow mailing list: https://lists.lfai.foundation/g/tacmlworkflow-generalcommittee

Participants

Working Group Lead: Ofer Hermoni <oferher@gmail.com>

Name

Affiliation

Email

LF ID

Ofer HermoniAmdocsoferher@gmail.com  Ibrahim HaddadLinux Foundationibrahim@linuxfoundation.orgJim SpohrerIBMspohrer@us.ibm.comNat SubramanianTech Mahindranatarajan.subramanian@techmahindra.comIan LumbSylabs.ioilumb@sylabs.ioJamil ChawkiOrangejamil.chawki@orange.comVishnu Ram OVMember of ITU FG ML5Gvishnu.n@ieee.orgAnimesh SinghIBMsinghan@us.ibm.comVijay R BommireddipalliIBMvijayrb@us.ibm.comGanesh HarinathVerizonganesh.harinath@verizonmedia.comKiril NejkovIFCknejkov@ifc.orgGordon I. MyersIFCgmyers@ifc.orgGuangji XueIFCgxue@ifc.orgHussain A. AlkazemiWorld Bank Group (WBG)halkazemi@worldbankgroup.orgPatrick FuGemini Open Cloudpatfu2005@geminiopencloud.comEdison Peng Gemini Open Cloudedison@geminiopencloud.comFitz WangTencentfitzwang@tencent.comClive CoxSeldoncc@seldon.io

Sophia Arakelyan

Simioticssophia@simiotics.com

Neeraj Kashyap

Simioticsneeraj@simiotics.com

Next Meeting

Thursday October 3, 2019 at 8:00 am PT.

Please contact Ofer Hermoni or email info@lfai.foundation to be added to the call invitation.

Meetings

How to Join:

Contact Ofer to get invites

Meeting Content (minutes / recording / slides / other):

DateMinutes

Meetings

Every 4 weeks on Thursday, 10am EST right after the TAC meeting.

To participate in meetings, please join the mailing list and subscribe to the group calendar: ML Workflow - Community Meetings & Calendar

Zoom: https://zoom.us/j/9918615568


Agenda for 06/04/2020 meeting:

  1. Start with Adlik project on interop
    1. document: https://docs.google.com/document/d/1_e7Ty9VvEXvgS-4YSuBbER8VA4_vnX6AtZE1bOw_Efc/edit?usp=sharing
  2. Map new project to ML workflow stack


Agenda for 05/07/2020 meeting:

  1. Kickstart of the interop work, identify top 1~3 use cases, champions that volunteer to lead the use case discussion, and milestones
    1. document: https://docs.google.com/document/d/1_e7Ty9VvEXvgS-4YSuBbER8VA4_vnX6AtZE1bOw_Efc/edit?usp=sharing
  2. Map new project to ML workflow stack


Past Meetings - 

Meeting Content (minutes / recording / slides / other):


DateMinutes
11 June 2020

Call was recorded.

View file
nameRecording Alejandro_Saucedo Interop_Seldon_Kubeflow 20200611.mp4
height250

Call was recorded, this is the record. And this is the chat.

  • We agreed to change the name of the sub committee from ML Workflow Committee to ML Workflow & InterOp Committee (MWI)
  • Zhipeng (Howard) Huang shared the slides of the InterOp Proposal and we agreed to move forward with the plan
  • The first use case we agreed on will be Federated Learning
  • We will create a slack channel for the MWI Committee
  • Zhipeng (Howard) Huang will lead the WMI Committee from now on as Ofer Hermoni is stepping down

  • ML Workflow stack (updated deck):
    • Ofer Hermoni shared the work was done off line with Marquez team to map it to the Data Consistency block
    • Jnu Gu of Zilliz provided the mapping to the Serving box
  • Interoperability proposal (see deck)
    • Howard Huang of Huawei presented a proposal for interoperability between the different LFAI projects and also external projects
    • We had a discussion if there is a need for a real interoperability effort or maybe a standard will be enough? We agreed that there is a need for a real interoperability effort
    • Next steps:
      • In the next ML Workflow meeting Howard will present a plan for the effort
      • In one of the next TAC meetings Howard will share the proposal with the broader team  

 

Call was recorded, this is the record. And this is the chat.

In the meeting we covered the following:

  • We reviewed the current version of the ML Workflow, and tried to identify the correct blocks for the two new projects (Milvus and Marquez)
    • Jun of Zilliz will think about that for Milvus
    • Ofer Hermoni will approach the WeWork team to identify the right location for Marquez
  • Project integration:
    • Natarajan Subramanian covered the status of the collaboration / integration between Acumos and Angel, and Acumos and Adlik. Due to the Corona epidemic everything is slowed down currently in China
    • Natarajan Subramanian also mentioned the work Acumos is doing with AI360. Currently in discussions phase
  • Jessica Kim shared the status of the integration infrastructure Huawei has in Hong Kong, and the plan to add more locations based on Tencent public cloud - both in China and Canada
  • Jessica Kim suggested to leverage the integrations and the ML Workflow activity as part of the outreach activity

 

In this meeting we had a team from RedHat to present ODH (Open Data Hub)

  • In this meeting we started recording our calls. The Zoom recording can be find here. The chat here
  • Deck presented is available here
  • Ofer Hermoni gave a background about the ML Workflow working group goals and activity
  • Redhat team provided a presentation about ODH
  • Apparently there are many overlaps between the two initiatives
  • Next steps
    • Invite the ODH team to present to the TAC
    • A representative of the TAC will join one of the ODH meetings to present LFAI and ML Workflow

 

Agenda for the meeting:

1. Update about the integration between Acumos and other LFAI projects - Angel and ONNX
2. Review the latest version of the ML Workflow slides and map
3. Discuss our focus going forward
Agenda for the meeting:

Minutes:

  • We have now a dedicated mailing list, please make sure you register so you can get all the emails and invites
  • Ofer Hermoni shortly explained to goals of the committee
  • We mapped the new project Sparklyr on the ML stack, and updated the deck
  • Natarajan Subramanian and Fitz Wang updated regarding the integration between Angel and Acumos - the technical teams are working together on the integration
  • Natarajan Subramanian updated re the status of the integration between Acumos and ONNX. Jim Spohrer said he will dedicate more resources to help with that
  • Logistics:
    • Everyone should register to the mailing list in order to get the invites
    • Meetings will be scheduled for once in 4 weeks on Thursdays 10am EST (immediately after the TAC call)
  • We discussed the goals of this committee:
    • The integration between the different projects will be done by the technical teams of the projects. This committee will support that and identify potential collaboration opportunities and identify resource gaps
  • Jim Spohrer suggested to work with the Red Hat team on the Open Data Hub initiative. He will invite Sherard Griffin to talk in the next meeting
  • Jim Spohrer encourages collaboration and integration to IBM AI Fairness framework, it will help to convince IBM to contribute this framework of projects to LFAI

 

  • Goals defined for the meeting:
    • Reevaluate the ML Workflow stack according to feedback we got
    • Work on integration between different LFAI projects
  • Ofer Hermoni shared that there is a lot of interest from the community (LFAI members and the broader community), and many joined this effort, we have 18 people representing 8 companies!
  • evaluate the ML Workflow stack (see here updated deck):
    • Patrick Fu (from Gemini Open Cloud) shared their view on the ML Pipeline (see slide 2 on the deck), main difference was the data consistency layer - we added it to our stack
    • Ofer Hermoni mentioned that Fitz Wang showed in OSS that Angel supports Feature Engineering as well - we added that to the stack
    • Next step - update existing projects on the new stack
  • Integration between different projects:
    • Natarajan Subramanian shared with the team the work we are doing regarding the integration between Acumos and Angel and between Acumos and Horovod
    • Next step - set a "1:1" session dedicated to the integration between Angel and Acumos (Thursday September 19th)
  • More next steps

 

  • We farther refined the presentation and discussed next steps for this group
  • IBM presented some of the internal work they are doing around AI open source

 

  • We discussed the different open source projects and mapped them to our stack. See the results here
  • Next week Animesh will present a framework IBM is building and we will discuss how it can be beneficial to create collaboration between different LFAI projects
  • We will also prepare content for the next TAC meeting in which we will review our progress

 

  • We defined our goal, and started to review the stack. This is the result


Participants (archive as we moved to use the mailing list)

Committee Lead: Zhipeng Howard Huang Zhipeng (Howard) Huang

Name

Affiliation

Email

LF ID

Ofer HermoniAmdocsoferher@gmail.com  
Ibrahim HaddadLinux Foundationibrahim@linuxfoundation.org
Jim SpohrerIBMspohrer@us.ibm.com
Nat SubramanianTech Mahindranatarajan.subramanian@techmahindra.com
Ian LumbSylabs.ioilumb@sylabs.io
Jamil ChawkiOrangejamil.chawki@orange.com
Vishnu Ram OVMember of ITU FG ML5Gvishnu.n@ieee.org
Animesh SinghIBMsinghan@us.ibm.com
Vijay R BommireddipalliIBMvijayrb@us.ibm.com
Ganesh HarinathVerizonganesh.harinath@verizonmedia.com
Kiril NejkovIFCknejkov@ifc.org
Gordon I. MyersIFCgmyers@ifc.org
Guangji XueIFCgxue@ifc.org
Hussain A. AlkazemiWorld Bank Group (WBG)halkazemi@worldbankgroup.org
Patrick FuGemini Open Cloudpatfu2005@geminiopencloud.com
Edison Peng Gemini Open Cloudedison@geminiopencloud.com
Fitz WangTencentfitzwang@tencent.com
Clive CoxSeldoncc@seldon.io

Sophia Arakelyan

Simioticssophia@simiotics.com

Neeraj Kashyap

Simioticsneeraj@simiotics.com