Asset Repository WG - 20 July 2021

Attendees

[x] Michael Johnson (Apple) + WG Chair

[ ] Joshua Minor (Pixar + OpenTimelineIO)

[x] Eric Enderton (NVIDIA, DigiPro)

[x] Sean McDuffee (Intel)

[ ] Eric Bourque (Autodesk)

[ ] Will Telford (Autodesk)

[x] JT Nelson (Pasadena Open Source consortium / SoCal Blender group)

[ ] Sergio Rojas (Arena World)

[x] Scott Dyer (AMPAS)

[ ] John Mertic (Linux Foundation)

[ ] Nick Porcino (OTIO, OpenEXR / Pixar)

[ ] Lee Kerley (SPI)

[ ] Phil Sawicki (Autodesk)

[ ] Roman Zulak (Imageworks)

[ ] Orde Stevanoski (Imageworks)

[ ] Cary Phillips (ILM, OpenEXR)

[ ] Sebastian Herholz (Intel)

[ ] Erik Hansen

[ ] Sam Richards (Disney Imagineering)

[ ] Carson Brownlee (Intel)

[ ] Haley Kannall (Amazon Web Services)

[ ] Jens Jebens (Animal Logic)

[x] Darin Grant (Animal Logic)

[x] Matthew Low (Dreamworks Animation)

Regrets:

Jens Jebens (Animal Logic)

Agenda

• Proposing a formal Asset Repo project
• How are contributions approved for inclusion?
• How are modifications and extensions approved?
• Platform - Github or what?
• TSC, roles, governance
• List of committers
• Application form here

Notes

• Platform: Github, tar-balls, other?
• Both files and tar-balls?
• Large binary assets require Git LFS, which is another layer of tools.
• Don’t know a good open-source alternative.
• Git LFS: large file transfers are expensive. Change the README at the top of the zip file, costs for transfer of whole thing.
• AL assets will be on AWS. Pushing at Siggraph. Press release. FX Guide article. Home page on AL web. We can have corresponding home page on ASWF wiki, when we’re ready. Happy to mention ASWF and the Asset Repo License in publicity.
• Moana broken into three packages.  AL could be similar, 1. preview version 2. full textures, etc.  Moana has the small text files (README, license) separate as well, which is smart.
• “Guided tour”: Could be text, or Youtube video of a live interview.
• AL plans to be active in taking suggestions, adaptations to other renderers, etc. More additions, as permitted by gaps in production schedule.
• Moana USD delay: partly updating to new USD versions. Want to align with VFX Platform….
• Lights, materials, that part of USD is in flux. Basics of 3d scene is more stable.
• Asset Repo Platform
• README can have links to data (tar balls), and any other resources (video tours, …).
•  Links can go to whatever cloud service. Can migrate as needed.
• Name
• Call it data-sets? Nah, assets.
• Call it Motion Picture Assets? Nah, keep it simple. What about television, XR.
• We have 15-minute slot at Open Source Days. Can say whatever donors (Animal Logic, Intel, Amazon, ...) want us to say, about contributed assets. Michael and Eric will host Q+A afterwards.
• Welcome, Matthew: Dreamworks looking at donating a collection of assets representative of pipeline structure, in USD.
• Running license through Legal. Negotiating asset versions with Production. After Siggraph.
• How do you do references, payloads, proxy vs render, variant sets, etc. Not massive assets. Won’t run with RenderMan out of the box since it’s DWA pipeline. Can evolve to support usdview / Houdini / etc.
• Thank you - Project Proposal now ready. Send suggestions.

Asset Repository WG - 6 July 2021

Attendees

[x] Michael Johnson (Apple) + WG Chair

[ ] Joshua Minor (Pixar + OpenTimelineIO)

[x] Eric Enderton (NVIDIA, DigiPro)

[ ] Sean McDuffee (Intel)

[x] Eric Bourque (Autodesk)

[x] Will Telford (Autodesk)

[x] JT Nelson (Pasadena Open Source consortium / SoCal Blender group)

[x] Sergio Rojas (Arena World)

[x] Scott Dyer (AMPAS)

[ ] John Mertic (Linux Foundation)

[ ] Nick Porcino (OTIO, OpenEXR / Pixar)

[ ] Lee Kerley (SPI)

[ ] Phil Sawicki (Autodesk)

[ ] Roman Zulak (Imageworks)

[ ] Orde Stevanoski (Imageworks)

[ ] Cary Phillips (ILM, OpenEXR)

[ ] Sebastian Herholz (Intel)

[ ] Erik Hansen

[ ] Sam Richards (Disney Imagineering)

[ ] Carson Brownlee (Intel)

[x] Haley Kannall (Amazon Web Services)

[x] Jens Jebens (Animal Logic)

[x] Darin Grant (Animal Logic)

Regrets:

Sean McDuffee (Intel)

Notes

Animal Logic scene

Two versions

1- Vanilla USD, std usdpreview shaders

2- Animal Logic version, with new ideas / USD proposals baked in

  e.g. Render Procedurals - access geom from the stage

3- Production quality textures (from 100MB to 20 GB)

Demonstrate all aspects of USD

Also need demo stuff, training materials … Don’t own customer assets.

Expected updates, after initial donation?

  Maybe add ideas, like Render Procedurals

  Pull requests for other renderers, sure

    Not sure github is where you want textures

If AL will put it up somewhere with a password, we’ll get some TAC folks to look at it

  Need to clear by AL lawyers … Then TAC readers clear with their lawyers …

  Easiest thing, put it under Asset License, since that’s already approved.

Transition this Working Group into a, what, TSC or standing group

Posting at Siggraph -

Have AWS sponsor, if ASWF can’t post by Siggraph, AL will post there.

Great to use this as forcing function!

1950s workbench / lab. Add dry ice later?

Expensive to create.

Documentation started. Here’s what it is, what it isn’t. Guided tour to the assets.

Versioning and format

  Gzip / tar files with explanatory text ?

  Forum / slack channel?  Discussion, questions, python scripts people came up with …

  Or allow discussion elsewhere - USD interest group, CG Talk … Reddit …

  Lower the burden on donor: Community could ask a few questions, create documentation?

    E.g. Autodesk QAs could make an Arnold version

  Review process for enhancements

  Mental model of few/rare updates, rather than continuous PRs

  Although this is more live than say Moana (movie’s done) - evolving teaching/training document

Michael uses two representations. One is a Github repo with each layer a file (often ASCII). Tarball harder to point to a file. Easy to name, find, and dump a file with organization of layers of the scene. WBN to have links to individual files.

  How many files? About 1,000.

  USD Working Group can advise. But wanted donor’s preferences, first.

If someone makes an Arnold version, a Karma version

  Go back to donor, ask if it’s ok to put that next to the original?

Let’s just do it, see how community responds.

Exciting!

Also helps amplify ASWF - artists want to help, now they can contribute.

Apple porting Storm to Metal, integrated into Preview. Want to see someone else’s USD file on a MacBook Pro :)  Current demo is Pixar Antiques Mall; just display color and opacity.

AWS Legal has notes on license (ok with contribute; notes on consuming); send to League of Lawyers.

Asset Repository WG - 22 June 2021

Attendees

[x] Michael Johnson (Apple) + WG Chair

[ ] Joshua Minor (Pixar + OpenTimelineIO)

[x] Eric Enderton (NVIDIA, DigiPro)

[x] Sean McDuffee (Intel)

[ ] Eric Bourque (Autodesk)

[x] Will Telford (Autodesk)

[x] JT Nelson (Pasadena Open Source consortium / SoCal Blender group)

[x] Sergio Rojas (Arena World)

[x] Scott Dyer (AMPAS)

[ ] John Mertic (Linux Foundation)

[ ] Nick Porcino (OTIO, OpenEXR / Pixar)

[ ] Lee Kerley (SPI)

[ ] Phil Sawicki (Autodesk)

[ ] Roman Zulak (Imageworks)

[ ] Orde Stevanoski (Imageworks)

[ ] Cary Phillips (ILM, OpenEXR)

[ ] Sebastian Herholz (Intel)

[ ] Erik Hansen

[ ] Sam Richards (Disney Imagineering)

[ ] Carson Brownlee (Intel)

[x] Haley Kannall (Amazon Web Services)

[ ] Simran Spiller

[ ] Dan Rolinek

Regrets:

Joshua Minor (Pixar + OpenTimelineIO)

Eric Bourque (Autodesk)

Notes:

ASC StEM2 Assets

• Michael: License conversation going well. Volunteer organization, no staff lawyer, no staff sys admins.
• Haley: AWS can help with hosting, and Cloud Watch / robot guards.
• Stem 1 - https://www.dcimovies.com/2014_StEM_Access/
• Let’s figure out the $ ask, take to ASWF Board.
• People don’t normally need to download the entire thing. Downloading entire thing would boost the cost.
• ASC has Azure commitment for 1st year, starts September. We could start as silent mirror, one of our pointers like Moana, Netflix. Pick it up if/when Azure ends.

Cost estimates

• Sean: 1-2 TB could be $100/month for simple S3 bucket.
• Ask any repos you know: Size, stead-state downloads, burst downloads.
• ~1400 downloads? Reasonable number.
• Michael: gitlfs for large files, kinda tough.
• Version control? S3 has none? Check into S3 Version. We’ll probably want something. Moana did get fixes.
• USD version promised 1.5 years ago; still not out? Like NVIDIA said, this stuff is hard; we shouldn’t expect we’re going to do lots of refinement before posting a donation.
• “De-pipelining” assets from a film.
• Contribution standard is, “any publicly readable format” - some way to load it with widely available software. Ideally free - e.g. student version of RenderMan.
• Haley: “Spanner”: Maya, Arnold, yeti for hair (discussed migrating to xgen) - you might need the right version of all these. JT was able to get it working.
• Maya files. Lighting files are Alembic.
• Could deliver final Maya lighting file.
• USD varies from DCC to DCC, but mostly works, maybe sometimes subdivs just show the caches.
• FBX is OK, but different importers get different amounts. Alembic is pretty portable. Except sometimes you don’t get creases.

License converging, board members signing off.

Wish list needed.

Content offers:

• Amazon, ASC: above.
• Intel: Size? Clouds should be minimum 200^3, full library 2 TB maybe? Sean will send around a catalog. We can reach out to OpenVDB TSC.

• Intel: Also have a full scene with textures, < 16 GB, USD. Plan to release under the new license when finished.

Next step: Try to get timeline for contributors to tidy up assets enough to post.

• Intel: Clouds ready now.

For project-related assets, ask that project’s TSC.

For non-project e.g. general 3D assets like Spanner, ask the TAC.

Asset Repository WG - 8 June 2021

Attendees

[x] Michael Johnson (Apple) + WG Chair

[x] Joshua Minor (Pixar + OpenTimelineIO)

[x] Eric Enderton (NVIDIA, DigiPro)

[x] Sean McDuffee (Intel)

[x] Eric Bourque (Autodesk)

[x] Will Telford (Autodesk)

[x] JT Nelson (Pasadena Open Source consortium / SoCal Blender group)

[x] Sergio Rojas (Arena World)

[x] Scott Dyer (AMPAS)

[ ] John Mertic (Linux Foundation)

[ ] Nick Porcino (OTIO, OpenEXR / Pixar)

[ ] Lee Kerley (SPI)

[ ] Phil Sawicki (Autodesk)

[ ] Roman Zulak (Imageworks)

[ ] Orde Stevanoski (Imageworks)

[ ] Cary Phillips (ILM, OpenEXR)

[ ] Sebastian Herholz (Intel)

[ ] Erik Hansen

[ ] Sam Richards (Disney Imagineering)

[ ] Carson Brownlee (Intel)

[x] Haley Kannall (Amazon Web Services)

[ ] Simran Spiller

[ ] Dan Rolinek

Regrets:

Notes:

• Wish List

• One from Josh / OTIO.

• Need more.

• License legal conversation

• Proceeding on legal mail list. Draft converging (we hope). Michael posted latest draft on Slack.
• Getting near “last call”.
• “Product demonstration” is tricky. Line between demonstration and marketing.
• Talk at Siggraph - sure
• Social media campaign showing performance results - maybe?
• Can’t “imply endorsement”.
• In general, ask your lawyer.
• EricB: Lawyers might not let marketing post anything on social media.
• Show Moana in blog post?  Gray area, my blog post yes, Marketing’s blog post maybe not…  Intent to inform.
• If you’re talking about your product, is that permitted by “software and hardware product demonstration”?
• Turntable of Moana Island OK.  Clip of Moana film, not OK.
• Hosting
• AWS pretty cheap, say 250 GB under $100/mo.
• Sean making request via LF web page.
• AWS eager to assist - talk to Haley.
• Are there good strategies to avoid bots sucking bandwidth? (Say we have 100 TB …) Haley will check into it.
• Want it to be popular. But want bandwidth to go to a good cause.
• ASC STEM2 project. See description in Slack thread.
• 6 - 20 TB. Creative Commons.
• Could we host?
• Let’s understand the pieces of it.
• Overlaps with Netflix open content?
• OTIO - Darby making some progress on timelines that point into the Netflix content.
• ASC a more neutral party. Netflix more strictly “production”.
• Once we have footage / EXRs:  People will transcode. Coordinate with Transcoding / encoding best practices that Sam Richards is talking about. “Here’s the best MPEG settings.”
• And question of, where do transcoded versions live.
• Do we host the transcoded version, or the recipe to create it, or neither.
• Cherry pick a couple good ones, to show off our repo.
• AWS has short film, releasing assets on Github, with tutorial. Simplified for Github size restrictions. Would like to release full res.
• 2 char rigs
• massive airship
• environ, cloud sims
• Haley will post film and tutorial in Slack
• Would be interesting to know total size
• Clouds are couple TB
• Apple photogrammetry pizzas now available as USD 🙂  🍕

Asset Repository WG - 25 May 2021

Attendees

[x] Michael Johnson (Apple) + WG Chair

[ ] Joshua Minor (Pixar + OpenTimelineIO)

[ ] John Mertic (Linux Foundation)

[ ] Nick Porcino (OTIO, OpenEXR / Pixar)

[x] Eric Enderton (NVIDIA, DigiPro)

[x] Sean McDuffee (Intel)

[ ] Eric Bourque (Autodesk)

[ ] Erik Hansen

[x] JT Nelson (Pasadena Open Source consortium / SoCal Blender group)

[ ] Lee Kerley (SPI)

[ ] Phil Sawicki (Autodesk)

[ ] Roman Zulak (Imageworks)

[ ] Orde Stevanoski (Imageworks)

[ ] Cary Phillips (ILM, OpenEXR)

[ ] Sebastian Herholz (Intel)

[ ] David Morin (ASWF)

[ ] Sam Richards (Disney Imagineering)

[x] Will Telford (Autodesk)

[ ] Carson Brownlee (Intel)

[x] Sergio Rojas (Arena World)

[ ] Haley Kannall (Amazon Web Services)

[ ] Simran Spiller

[ ] Dan Rolinek

[x] Scott Dyer (AMPAS)

Regrets:

Joshua Minor (Pixar + OpenTimelineIO)

Notes:

• License legal conversation
• Dreamworks proposed some clarifications
• Michael will check with Board on these
• Wish list - Soliciting requests
• Michael to mail project leads list
• Michael to post question on project Slack channels
• Community requests. Filter through TAC?
• Back end, costs
• Sean talking to LF. Will send out pricing structure.
• 250 GB to open? 1 TB eventually? Moana around 90 GB.
• Larry’s wish list might be deep full-color images, volumes. Those get big.
• Downloads are bursty. TBs in first couple weeks, then tapers quickly.
• Netflix has made available nice finished EXRs. OTIO talked about building a small asset that points to Netflix EXRs.
• Would Disney or NVIDIA move an asset to ASWF, and point to it? Or we point to them?
• Intel VDBs: 20 cloud examples at low/med res might be 40 GB altogether. Could make them quite large.
• Michael’s USD backlot is 290 GB.
• Allocating that was heavier lift than expected.
• Robots download too much. Bot blocking helps. Captchas or emailed links (“magic link”). JT’s VR projects do this, Disney Moana does this.
• Ask our vendor for best practice.
• Consent to license.
• Publicity. How to get started. Can start small / soft launch.
• Can we get license, scale estimates, and back end in place by Siggraph?
• Announce at Open Source Days.  DigiPro, The Pipeline Conference.  AMPAS Sci Tech committee.
• Want to reach studios and other potential donors.
• ASC, other professional groups?
• Will: Mental test, looks at Links to Open Assets. Would this license work? Would this process work?
• Soft launch
• Will: Don’t need biggest assets first; go for quality assets.
• Downloads will tell us what our bandwidth scale will be.
• If your machine isn’t huge, you might download multiple times! Blow away to make space. Fiber makes it pretty easy.
• Process for acceptance?
• Talk to TSC of relevant project. Look for their support. Else, send a standardized respectful rejection letter giving the reason.
• Don’t need to formalize this process yet.
• Publishing a spec would help. Version the spec. Helps people know what they’re downloading also.
• Project TSC will have to help here.

Asset Repository WG - 11 May 2021

Attendees

[x] Michael Johnson (Apple) + WG Chair

[x] Joshua Minor (Pixar + OpenTimelineIO)

[ ] John Mertic (Linux Foundation)

[ ] Nick Porcino (OTIO, OpenEXR / Pixar)

[x] Eric Enderton (NVIDIA, DigiPro)

[x] Sean McDuffee (Intel)

[ ] Eric Bourque (Autodesk)

[ ] Erik Hansen

[x] JT Nelson (Pasadena Open Source consortium / SoCal Blender group)

[ ] Lee Kerley (SPI)

[ ] Phil Sawicki (Autodesk)

[ ] Roman Zulak (Imageworks)

[ ] Orde Stevanoski (Imageworks)

[ ] Cary Phillips (ILM, OpenEXR)

[ ] Sebastian Herholz (Intel)

[ ] David Morin (ASWF)

[ ] Sam Richards (Disney Imagineering)

[x] Will Telford (Autodesk)

[ ] Carson Brownlee (Intel)

[ ] Sergio Rojas (Arena World)

[ ] Haley Kannall (Amazon Web Services)

[ ] Simran Spiller

[x] Dan Rolinek

Notes:

• Intel license modifications look great. Simple, clear.
• Sent out to League of Lawyers, couple days ago.
• We want to say to donors “This is the license.”
• More permissive license - like public domain - also ok
• If something’s donated under public domain, will lawyers want us to re-license it to our standard license?
• So let’s go to next phase, get something to put under that license.
• Projects’ wish list
• EXR: Metadata examples. Deep files, real world full color.
• Trying to ballpark sizes. Sean pinging Kimball for EXR sizes.
• OTIO:  Do you need 1 complex example, rather than many small ones?
• Dream case: Full feature film, in midst of production, with max clips and audio tracks. Like what Josh sees on his work desktop right now. :)
• Blender: We’ll find out what their process is, do they edit in Blender, etc.
• Whole film - size? 50 MB for composition, but media size depends on resolution etc. Can measure Netflix example.
• Does OTIO need image quality? No.  But Playback + Review, other groups, will need high-res. Large quantity to scrub through is a great benchmarking case.  Also Premiere / Composer / SW vendors could use as real as they can get.
• OTIO goes through different departments - edit, color-grade, etc. Useful for an asset to identify where it is in the workflow.  May want to test other workflows, load the asset differently.
• Netflix has media on line. We could make OTIO that points to those remote media; that would be small. Multi project overlap.
• Sean in touch with Andy Greenberg[?] @LF to set up hosting.
• Sean talked to Nick about DIsney hosting. On S3. 70-80 TB of downloads in first two weeks (!). He found S3 costs to be “manageable”.
• Peaky load can be handled by bit-torrent and other peer-to-peer.
• “Peertube” has torrent built in.
• Want ASWF branding. Not just, someone Googles it and finds it on another mirror site.
• Clear labeling: Say up front, this is a 1 TB download.
• AWS is premier member of ASWF … We can go to them with estimated sizes and access patterns.
• Do we need to discourage looky-loos / random non-industry hobbyists? Is it $15K versus $7K, or is it $200K versus $15K?
• Prefer “intentional” downloads. And not bots. 
• From Slack: Morten Vassvik: Hosting VDBs a headache. “Just the small amount of animations in the link provided by JT above completely dominated our traffic the first month. I think we hit 10 TB of traffic or something like it for that month. These days we simply host it on Mediafire, which is fine in practice, but a bit sketchy as it's serving ads after a certain number of downloads. At least it doesn't cost us a fortune for now.”
• Soliciting wish lists: Michael to email TAC.
• Then what. Socialize with TAC and Board. Create template language around what does a request look like, what does a submission look like.
• Vendor requests could go through the project. Requests from the community.
• If a Project rejects - or is silent - about a contribution, then we decline and say “sorry, too big for too few users for us; if you want to put our license on it, we’re happy to link it on our list page”.
• Could imagine OpenVDB being 70% of the budget for 5 examples :)
• How was VFX Reference Platform announced? Maybe we should announce at Siggraph? VFX Ref is connected to VES; we can announce through that. Certainly announce at ASWF Open Source Days. Siggraph’s list of BOFS is scoured by lots of people; VFX Ref Platform got listed there.
• Last year you had to pay $100+ to attend the BOFs. So some groups didn’t hold a BOF.
• BOF listing is like an elevator pitch. Brief.
• Let’s get something announceable together by Siggraph. Good to have a deadline...
• Would NVIDIA consider moving ORCA to ASWF?  Not sure it’s “film-like” enough to be accepted.
• ASWF started partly for open source projects that didn’t have a solid home. Want to give assets a solid home, too.
• Josh to send license to USD, propose to re-license the USD sample assets.

Asset Repository WG - 27 April 2021

Attendees

[Roll not taken]

Notes

Thoughts on what assets we accept

We have a license we need you agree to

• If we can’t/won’t take it, we can put you on our link list page and advise on license

We would ask ASWF projects to “sponsor” or vet the asset

Hosting options:

• Can we use bit torrent?
• How do we avoid DDOS?

Sean points out we should figure out the economics:

• Storage space
• Number of downloads

Need to get a wishlist from the projects

Different kinds of EXRs - w/AOVs and not

Need to consider versioning - can we just use dated folders?

How live do we think these assets will be?

How do we deal with forks of assets

Can we get numbers from places like ORCA?

Links to Open Assets

• Vendors who want stuff are not just projects
• “Badging”
• How do we maintain these assets? Who gets to contribute fixes? Are we asking the donor to do that? Are they going to allow it?

Action items

• Ask ASWF projects for wish list
• Ask ORCA for download info
• Ask Disney for Moana island download info
• Ask LF for guidance on hosting numbers
• Make a landing page for the wishlist from the ASWF projects

Size estimate

Asset Repository WG - 13 April 2021

No notes taken?

Josh: Brief update on Netflix Open Content assets, and their licensing.