Writing an MP4 Muxer for Fun and Profit

Having worked with some MP4 demuxing for my extension [1], I feel the pain. Lots of times I would play the video only to find inexplicable issues such as drifting audio. I highly recommend using an mp4 inspector tool, such as mp4box [2], to debug these issues.

1: https://github.com/Andrews54757/FastStream

2: https://gpac.github.io/mp4box.js/test/filereader.html

Nice, when playing around one weekend trying to see if I could use ipfs as a transport layer for streaming video I got hung up because most video formats I tried behaved very poorly with inconsistent streams where you may not have the beginning. I ended up on mpeg-ts as the best behaving of the bunch. It felt a little weird, as I was sort of expecting something more modern to have better performance, but seeing as my goal was not to evaluate video formats but just ship them around I just accepted it and moved on.

Thinking back on it now, I just did a little trial and error until I found something that worked, but what would I search for if I was trying to find data on how... ?streamable? an encoding is?

If curious, I got my proof of concept working but it was unpleasantly slow. I blindly chunked the incoming stream into megabyte sized chunks registered the chunks on ipfs then used ipfs pubsub to announce the chunk to any watchers. The watcher would watch the pubsub channel for announcements download the chunk and try to reassemble it in order and play it. one neat side effect that I found was when the stream was done if I had stored all the ipfs address I could then generate a whole ipfs file structure you could use to download the stream at a later date.

Can someone explain how does an existing media player understand the new mdat format without modification? I assume if they find a completed moov at end of the file, it would recognize the file as a unfragmented mp4. It should then try to find a list of recognized codecs directly inside the mdat (like in the first picture), but instead they will find another moov, a bunch of moofs and sub-mdats, all of which are clearly not proper for a unfragmented mp4. Why doesn't the player report this as a "unrecognizable, badly formatted" mp4 file?

This is awesome work. I’ve coded some extensions for mp4 livestream to handling dozens of real-time streams and I’d love to try out the multi stream mux / demux…

> It kind of hurts that several days of work and research can be summed up in a couple paragraphs, but that's what the "pain" part in the subtitle is for.

Having recently written my own fragmented-MP4 remuxing library, I felt this pain too, and my soon-to-be-published writeup has very similar things to say about the ISO's paywalling practices.

I think one of the hardest parts of ISO-BMFF, aside from spec availability, is that it's pretty hard to implement "cleanly", making existing code confusing to use as reference. (My own implementation is certainly not clean either)

It looks like GStreamer has supported this for a few years: https://gstreamer.freedesktop.org/documentation/isomp4/GstBa...

I always forget about GStreamer but I think I have a perfect application for it. Hopefully it’s easier to use as a library than MediaFoundation or FFMpeg.

MP4. The answer to the question of "Is there a way to make RIFF and AVI even worse somehow?" It makes you genuinely pine for MPEG2 Transport Streams. ISO 13818 for life.

Great work!

Would love to see MP4 Hybrid supported in popular packages like mp4-muxer [1] and mp4box [2] someday.

1: https://github.com/Vanilagy/mp4-muxer 2: https://github.com/gpac/mp4box.js

> moof (Movie Fragment Box)

Very cute easter egg. Moof is what dogcows say: http://clarus.chez-alice.fr/history.php

So this is a “soft” sequential access limitation (we can tolerate some random writes to data as long as it is small enough and short enough). I wonder if there are formats that result in finished indexed multimedia file with “hard” sequential access, when nothing can be overwritten.

(context, this is talking about fragmented MP4 downsides)

> 2. They are slow to access on HDD or network drives, as each fragment's header needs to be read to get the complete metadata of the file and start playback

Huh? That's not right. The whole point of fragmented MP4 is that you can access any fragment without having to read the headers of the other fragments. That's why adaptive streaming is built around fragmented MP4.

> Except there is no profit, only pain

I have 20 years or professional experience and my conclusion, if someone asked, what IT boils down to: pain.

The pain is what filters who can succeed and who fail. Can you endure hunting a bug for 7 hours in your chair? Can you fix problem after problem to get a system running? Everything that can fail, will fail, and you have to deal with it.

"The new MP4 output now also supports multiple video tracks"

MP4 has been able to have multiple video streams for quite some time. One of the very first advanced MP4 authoring tools I saw in the early 00s allowed for this, and we used it to make a few advanced files to demo the "new" MP4 format. Much like multi-angle DVDs, this was a niche feature that did not gain very much attraction. I could see why someone not around at that time might think this is a new feature, but it's not