Re: [media-workshop] [DRAFT] Report of the Media Controller workshop in Espoo, Finland – July, 29-31 2015

Hi Mauro,

Thank you for the report. Please see inline for a couple of small comments.

On Wednesday 12 August 2015 10:11:45 Mauro Carvalho Chehab wrote:

Hi,

This is the first draft of the Media Controller Workshop report.

It was complex to do this report based on the notes we took, as some notes were incomplete. I did my best, but I'm sure some items need some review, in special the presentations.

Feel free to review and send me your comments for me to add in the final MC workshop report and send to the main mailing list.

Sakari,

I don't have the email of the two guys from Intel that participated on our first day. Could you please send it for me to complete at the report?

Thanks! Mauro

--

Report of the Media Controller workshop in Espoo, Finland – July, 29-31 2015

This is the first workshop dedicated to the Media Controller. We actually did a v4l summit in 2010, also in Finland, that stablished the current foundation for the media controller, and was meant to satisfy the needs of properly reporting the pipelines on the SoC used by smartphones. The focus of this year's workshop is to clarify the kernel→userspace interfaces and extend the Media Controller to be used on other subsystems that need to represent graphs, like DVB, ALSA and IIO.

[snip]

3. Discussions about the Media Controller API and the needs to extend it to

support the requirements for DVB, ALSA and IIO

3.1) Discussions took at July, 29

Basic requirements:

• Properly represent kernel→userspace API interfaces;
• Allow dynamic pipeline changes;
• Define the namespace for interfaces and entities;
• Support additional properties;
• Better represent entities that belong to more than one type

It was proposed to use the concept of planes, as defined by ITU and IEEE to represent different layers of a data or media network (ITU-T G.800). So, we would have control and data planes.

http://linuxtv.org/downloads/presentations/mc_ws_2015/media_controller_summi t_needs.pdf

This is a concept largely used on networks, as show at: http://etherealmind.com/controller-based-networks-for-data-centres/

Multiple options to express association between device nodes and entities: - Same IOCTL API as for data links, difference made between data and control links - Different IOCTL API to enumerate interfaces, with associated entities - Properties

The decision was, instead, to add a new graph element type (Interface) to represent the elements that controls the entities.

It was agreed with the following terminology: - Entity: Functional unit, typically a hardware component or subunit in a hardware component - Pad: Data Input/Output of an entity - Interface: Software control access to a entity (e.g. v4l2-subdev)

"to an entity" is a bit restrictive here, as an interface can be used to control several entities. I would instead define interface as "control point implementing a userspace API" or something similar.

- Link: Connection between a source pad and a sink pad

It was also proposed to use the name “association” for the links between interface and entity, but there was no consensus about that idea.

EDITOR'S NOTE: It was decided on July, 31 to use the name “links” also for the connection between an entity and an interface. We used the term “interface links” for such links when we need to distinguish them from “data links” (e. g. pad to pad links).

Was it ? I thought we hadn't reached an agreement on that :-) I'm not advocating for the name "association" though, but I thought the point has just not been settled yet.

Mauro's concern is that there will be lots of code duplication in the Kernel if we use a different graph type for the interface links.

The relationship between Interfaces and entities are n-n: - One interface can control multiple entities - One entity can be controlled by multiple interfaces

Media interfaces - A concept of user space facing interfaces - An object of its own right (such as media entities) - At the same level than entities

What do you mean by "at the same level" ?

Working proposal (as proposed on July, 29)

- Introduce struct media_interface as an object type alongside with
  entities in the Kernel and in the user space
- See how the patches will look like; start with V4L2 and proceed with
  DVB
- How much will there be code duplication?
- Target 4.3 at earliest, more likely 4.4

The current API does a crap job on properly defining entities that are coupled to interfaces, and prevented the enablement of the MC DVB support, submitted in December, 2014. That needs to be fixed as soon as possible, as it is causing bad impact on using MC even for other V4L2 device types, like radio. So, Mauro stated that, as the MC API is currently on a broken state, no patches, drivers or framework changes related to the Media Controller will be merged upstream until this is fixed. Fixup patches may be merged, though.

- Hans would try to do the work (of proposing the userspace API) at the

week after the MC workshop.

Userspace API Requirements: - Enumerate Entities + pads/links (exists today) - Enumerate Interfaces + entity associations (and define which interface is the default one) - For entities: enumerate associated interfaces - Should support two application models: Entity-centric and Interface-centric

Additional API requirements that should be addressed in the future: - MC: 'versioning number' so apps are informed if the topology has changed. - MC: Notification events for topology changes

Points to solve: - entity.type - should be renamed to reflect what's there; - How to represent DVB entities/interfaces? - How to represent DVB network interfaces? - How to represent ALSA entities? - What fields are needed for media interfaces at the public API? - Changes at media-ctl userspace application

Some notes to help with network interface definitions: - man 7 netdevice - man 7 rtnetlink - Network interface index is constant for a network device - Network interface index can be converted to name etc. by an IOCTL

DRM subsystem uses u64 instead of pointers at the public API: - IOCTL design: http://blog.ffwll.ch/2013/11/botching-up-ioctls.html - Not widely used outside DRM - Most sub-systems have their own conventions

TODO: - Add graph topology version to the enumeration (and possibly other) ioctls

Question: should we redesign everything in order to support topology changes?

3.2) Discussions took at the afternoon of July30 and on July, 31st

It was discussed if we should add a generic graph framework in the kernel. This would require review on linux-kernel and such review may take a long time, as different requirements could pop up. So, perhaps not this time.

It was presented some topology for simple hybrid TV hardware (PC-customer hardware), at: http://linuxtv.org/downloads/presentations/mc_ws_2015/media_controller_summ it_TV_pipelines.pdf

An agreement was arrived for the UAPI interface.

What was agreed: media interfaces will be a separate graph element; links will be used to connect both entities (pads to pads) and media interfaces; PADs will use unique IDs preliminary header changes: http://linuxtv.org/downloads/presentations/mc_ws_2015/media.h

V4L2 applications need to know if they can capture from an input/tuner. To make that easy for them, it was proposed to add a new flag to VIDIOC_ENUMINPUT and whether the tuner is in use (and/or the tuner state) through VIDIOC_G_TUNER. This can't be done at the DVB side, however, as there are no reserved fields there.

But extensions to the DVB API are still possible, right ?

It was discussed about the need to allow bidirectional pads (needed for connectors like a coax that are bidirectional using time or frequency multiplexing)

TODO: Connector entities

It was proposed the following possible types of entities: Connector, DMA engine, processing block and controller block

Whather an entity is a connector needs to be told, as this can't be discovered

All others can be derived from other information already in the graph?

Entity functional properties

Most of the properties at entity level would be used to signal

capabilities that can be figured out using sub-system specific interfaces the application already uses

Interfaces

Agreement: interfaces do have a type
Types include V4L2, ALSA, DVB etc.
Sub-types are used to tell between different interface types inside the

sub-system

An interface can be a device node, network interface or a sysfs

directory

It was proposed two new ioctls: G_TOPOLOGY IOCTL should not be used to convey arbitrary sysfs paths G_INTERFACE to pass information on single interface; there, a single pointer set by the user would be feasible. The struct can be the same.

Wasn't the agreement that sysfs paths would be passed through properties ? That's at least what the media.h working copy seems to suggest.

Finding entities Especially device specific applications often need to find an exact entity the application requires This may be a certain scaler in the graph, for instance Configuration may involve e.g. private IOCTLs only a certain sub-device implements Name is only 32 bytes at the moment In practice, external entities have used i2c bus number and i2c address as part of the name This does not scale for DT based devices where the device is uniquely identified by a string that can be very long Blocks that are a part of an ISP there's been just a name of the block without any device specific information This causes that if one has multiple such devices in the system, the name is no longer unique The entity name is not enough to uniquely find an entity in a general case In a general case it is not possible to recognise an entity

Are those last two points about the current situation or about any API we could come up with ? If it's the former it should be clarified. If it's the latter I'm not sure to agree :-)

Hot pluggable buses and entities that have no serial numbers or
something equivalent
In order to best help the user space, all the information that matters
should be provided to the user

So, it ended by having device names such as "omap3isp resizer"

Other data could be used to enhance the device name, like: Bus Serial number Sysfs name (this may not be stable as sysfs has no official API stability, but may be still better than bus in some cases)

Doesn't it ? How about Documentation/ABI/stable/* ?

This should enable most of the use cases where entities need to be found

Properties

Array of properties as an IOCTL argument has the problem that the user would have to allocate enough memory for each key-value pair in the array. This is unfeasible.

The original proposal of using a text based format requires parsing by the user, albeit the kernel interface would remain simple

The original proposal wasn't using a text-based format to convey information between user space and kernel space. It used the text-based format for the purpose of discussing what properties should be, and left the question of the binary format for the ABI open.

The new proposal at the new media.h file (http://linuxtv.org/downloads/presentations/mc_ws_2015/media.h) combines the best of both: a single memory area amended with an array of properties that refer to the memory area

For the media properties, using strings the key type has benefits, as hierarchy is possible, with is not possible using an u32 integer for instance.

Should values be always strings or should other types be accepted? Probably having other types is a good idea.

Different options for arrays: As part of the key: entity.0/name

I would mention that this is just an example, other formats for array index as part of the key are possible (for instance "entity[0]/name").

As part of the property: add a length field. This is convenient except

for strings, where it works as well --- just count the nul characters.

Properties could be seen as re-implementing sysfs on the discussions with linux-api people. However they're much more light-weight. Also, there's no atomic access of all sysfs properties and sysfs is limited to page size Properties can be queried by media object, in which case the media object id would be provided The entire tree of properties can be obtained using object id 0

Another option for the kernel interface: key/length/value E.g. type (u32) | length (u32) | value (string) | key (string)

Media objects Media objects have an unique ID in the system Objects can be entities, pads or interfaces, for instance A few bits can be reserved for object types, e.g. entity == 0; interface == 1, pad == 2

Action Items:

• media_interface/media_topology: RFC for userspace API: Hans
• RFC patch series: internal implementation for interface/topology: Mauro
• Migration: add v4l-subdev media_interface: Laurent
• Migration: add explicit DMA Engine entity: Laurent
• media_properties: RFC userspace API: Sakari
• MC core support for dynamically adding/removing from the media graph:

Samsung

• Clarify/implement DVB network interface: Mauro
• MC support for Alsa: Lars: Need RFC before Alsa summit during ELCE
• MC support for DVB: Mauro
• MC default core support for interface-centric V4L2: Hans (?)
• MC support for IIO: Lars
• VIDIOC_ENUMINPUTS/G_TUNER: tell if the tuner/input is busy: RFC Hans,

Shuah has a backend for this.

• Fix media_device broken alloc/remove (devres): Shuah