drm: fix drm_mode_addfb() on big endian machines.

Userspace on big endian machhines typically expects the ADDFB ioctl
returns a big endian framebuffer.  drm_mode_addfb() will call
drm_mode_addfb2() unconditionally with little endian DRM_FORMAT_*
values though, which is wrong.  This patch fixes that.

Drivers (both kernel and xorg) have quirks in place to deal with the
broken drm_mode_addfb() behavior.  Because of this we can't just change
drm_mode_addfb() behavior for everybody without breaking things.  Add
the quirk_addfb_prefer_host_byte_order field to mode_config, so drivers
can opt-in.

Signed-off-by: Gerd Hoffmann <[email protected]>
Reviewed-by: Daniel Vetter <[email protected]>
Link: http://patchwork.freedesktop.org/patch/msgid/[email protected]
This commit is contained in:
Gerd Hoffmann 2018-09-05 08:04:43 +02:00
commit 6960e6da9c
2 changed files with 25 additions and 0 deletions

View file

@ -811,6 +811,20 @@ struct drm_mode_config {
uint32_t preferred_depth, prefer_shadow;
bool quirk_addfb_prefer_xbgr_30bpp;
/**
* @quirk_addfb_prefer_host_byte_order:
*
* When set to true drm_mode_addfb() will pick host byte order
* pixel_format when calling drm_mode_addfb2(). This is how
* drm_mode_addfb() should have worked from day one. It
* didn't though, so we ended up with quirks in both kernel
* and userspace drivers to deal with the broken behavior.
* Simply fixing drm_mode_addfb() unconditionally would break
* these drivers, so add a quirk bit here to allow drivers
* opt-in.
*/
bool quirk_addfb_prefer_host_byte_order;
/**
* @async_page_flip: Does this device support async flips on the primary
* plane?