DDS support

[RunDevelopment]

Hi. I'm the author of the DDS decoder PR and I wanted to talk about the future of DDS support in the image crate. I would like to work together to add support for reading and writing various DDS image formats to the image crate.

What's DDS

This section is meant as a refresher to make sure we're all on the same page.

DDS is container format for various compressed and uncompressed image formats. Its main purpose is to store textures for graphics applications (e.g. video games) as modern GPUs support DDS natively, allowing for the use of compressed textures directly on the GPU.

DDS also supports more than just simple images. Cube maps, 1D arrays, 2D images, 3D volumes, and arrays of all of the above (except volumes) are supported. And everything can have any number of mip maps. So a single DDS file can contain between 0 (empty array) and >2^32 images (if mip maps are included).

Furthermore, DDS supports a large number of image formats. The DXGI_FORMAT enum contains roughly 40 image formats that are relevant, but there are around 10 to 20 more that are used in old DDS files (D3DFormat). The complexity of these formats varies widely. Formats like R8G8B8A8_UNORM that can be "decoded" by just memcpy-ing bytes from disk, but formats like BC6 and BC7 require around 500-1000 LOC to decode a single block.

The present

Right now, DDS support is severely lacking. Only reading DDS files is supported, and only 3 (BC1/2/3) out the over 40 relevant DXGI formats are supported. Furthermore, those 3 formats are implemented incorrectly (*), causing noticeable discolorations for certain images.

(*) I'm saying "incorrectly", but they are technically spec-compliant. The spec allows +-3% error. Such an error is permissible for real-time graphics, where textures are samples millions of times per frame, but I believe that it is not suitable to allow such errors in a general-purpose image library.

Due to these short-comings, the usefulness of the DDS support in the image crate is currently very limited.

Given the comments on my old PR, there clearly seems to be interest in having the current situation improved, and I would like to help do just that.

What I have to offer

Over the last 2~3 months I've been working on a DDS de/encoder written in safe Rust. It currently supports 58 formats for decoding and 55 for encoding. The library currently has the following features:

• Parsing: It can parse the header of a DDS into a high-level representation of a DDS header (Header enum) and interpret that header to determine the exact layout of the data section (DataLayout enum). It can also detect and auto fix errors in the header, so old slightly incorrect DDS files can be read without issue.
• Decoding: Any of the supported formats can be decoded into a selected color format. So you can decode e.g. BC1_UNORM as RGBA_U8 or GRAY_F32 or similar. The library supports 12 color formats and will handle any necessary conversions. Of course, the native/original color format of the image format should be preferred. Decoding rects is also supported for all formats.
• Encoding: It's supported for all formats except BC6 and BC7. Images of any color format are supported for encoding (again, the library handles any necessary conversions). Error diffusion dithering is supported for some low-bit-depth formats. BC{1,2,3} also support a perceptual error metric based on Oklab for high-quality encoding.
• Speed: I benchmark and I optimize. Decoding a 4k texture takes around 100-200ms. The slowest format is BC6 with around 400ms. The other BCn formats are all around 50-100ms. Encoding is generally slower. Uncompressed formats take around 200-400ms for a 4k texture, but BCn formats are really slow. Encoding a 1k texture takes around 1-2sec for BC1 and around 300ms for BC4 on a single thread and at high quality settings.
• Correctness: There's lots of new code, so I have lots of tests. All formats are tested with snapshots, leading to around 400 snapshot images + many more snapshots, ensuring that all formats are correctly decoded and encoded. I also extensively compared the snapshots to DirectXTex by Microsoft. While my outputs aren't bit-by-bit equivalent for various reasons (mostly because DirectXTex likes to use fast approximations, while I don't), results are generally within 1 bit (u8) of DirectXTex with no visual difference. Of course, BC6/7 are bit-exact, as required by the spec.
• Few dependencies: I only use bitflags, zerocopy, and glam. Only bitflags is a public dependency. (Also, glam is only used for encoding. I just needed a small vector library, so it can be replaced if need be.)

I also want to say that I wrote everything from scratch. Honestly, I didn't think it would so much code, but it currently stands at ~12k LOC (not including tests and comments).

Note that everything is still under development, so please expect things to change and improve.

Collaborate

My goal is to use this library as a dependency for the image crate to power its DDS support. So I wish to collaborate to make it happen.

In particular, this is my first image library and my first crate to publish. So I would like to get feedback to 1) make the library more suitable for the image crate and 2) improve the library in general (I'm sure I made some suboptimal design decisions...).

I also want to say that I view the library in its current format more as a low-level primitive than a high-level DDS library. It can de/encoded single images with a DDS files, yes, but that's it. There's no high-level API de/encoding whole DDS files yet, and I haven't even thought about mipmap generation yet. So some pointers for making a high-level API would be appreciated.

Honestly, my ideal outcome would be to have the DDS library under the image-rs organization, so that it can be maintained and improved even when I, one day, don't have the time to do so anymore. You guys seem to know a little about maintaining image libraries :)

Please tell me what you think.

@HeroicKatora @fintelia

[fintelia]

I think this would be a great addition as a new crate under the image-rs org!

There's various aspects we'd have to work through before switching image over to using it, including making sure compile times aren't impacted too much, checking 32-bit and big-endian support, and so forth. But those don't need to block accepting the repository into the organization.

If you're interested, we can set things up so you retain write access to the repository with publishing permissions set to the image-rs/publish team.

[RunDevelopment]

Wonderful! Thank you for the quick response!

About the transferring the repo: I made the mistake of committing large files into it. The .git folder is currently at 35MB... I removed those files since, but the commits are still there. I was thinking about resolving this before the transferring the repo, but I'm not sure whether squashing everything into one commit and removing the history is the best approach (not that the history contains anything interesting). What do you think?

Also, should we rename it? I called it ddsd for DDS Decoder, because I originally only planned to write a decoder. But then I wrote half an encoder to test the decoder and thought "might as well." The name dds is still free on crates.io, so maybe we could rename it to that?

including making sure compile times aren't impacted too much, checking 32-bit and big-endian support, and so forth.

Definitely. Especially big endian has been worrying me correctness-wise, since I couldn't find a way to test it.

If you're interested, we can set things up so you retain write access to the repository with publishing permissions set to the image-rs/publish team.

Uhm, maybe? Write access would be nice, I think, but I'm not sure about publishing perms. I'm not saying my op-sec is horrible (at least, I don't think it is...), but it seems a little risky. Maybe I'm misjudging this, but I think it would be best if I just ask for those perms when the need arises.

[fintelia]

About the transferring the repo: I made the mistake of committing large files into it. The .git folder is currently at 35MB... I removed those files since, but the commits are still there. I was thinking about resolving this before the transferring the repo, but I'm not sure whether squashing everything into one commit and removing the history is the best approach (not that the history contains anything interesting). What do you think?

It should be possible to rewrite the git history to eliminate the large files while retaining the rest of the commits: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

(I'd recommend first making a copy of the entire repository to be safe.)

Also, should we rename it? I called it ddsd for DDS Decoder, because I originally only planned to write a decoder. But then I wrote half an encoder to test the decoder and thought "might as well." The name dds is still free on crates.io, so maybe we could rename it to that?

Either dds or image-dds seem like good names to me.

Especially big endian has been worrying me correctness-wise, since I couldn't find a way to test it.

This is the CI script we use to test 32-bit and big endian systems.

[RunDevelopment]

It should be possible to rewrite the git history to eliminate the large files while retaining the rest of the commits

Rewrite history it is then. I'll try to keep as many commits as possible.

Either dds or image-dds seem like good names to me.

image-dds is taken, so I guess dds it is then :)

This is the CI script we use to test 32-bit and big endian systems.

Oh, thank you! I'll set that up.

[RunDevelopment]

Alright. History has been rewritten (.git is 9MB now), CI for 32-bit and BE has been added (it found a BE bug in the test code), and I changed the name to dds. I also made issues for everything that still needs to be done (and a few future features).

So how are we going to do the transfer? Honestly, I assume that GH had something similar to collab invites where you would get a notification, but that's only for transfers between personal accounts and not organizations. I don't have perms to create repos in image-rs, so that didn't work.

So how are we going to do this?

1. I invite you as a collaborator, but I don't think collaborator can for transfers.
2. I can transfer the repo to you (personal account), and you transfer it to image-rs.
3. You can invite me to image-rs and give me enough perms to do the transfer myself.
4. Some other way I didn't think of.

Please tell me how you want to do this.

---

Quick workflow question: Are there any preferences around making branches in the repo directly or using a fork? I plan to switch to a PR-based workflow for my further development. Should I use a fork for that or no? I don't mind either way, just asking if there is a preferred way.

---

Also, the library is at a state where we could probably make a 0.0.1 release. I'm curious about your release workflow.

Then I could also work on a draft PR to use a library in image.

[RunDevelopment]

Any updates @fintelia?

---

On my side, I'm pretty much done. Over the last two weeks, I added a high-level de/encoder API, parallel encoding with rayon, and automatic mipmap generation with resize.

[fintelia]

Sorry, I've been quite busy. If you transfer the repository to my personal account I can transfer it onto the image-rs org.

I generally make PRs using a personal fork of the repository, though it doesn't make a huge difference.

Release workflow is pretty much:

1. Create a PR adding a changelog entry and bumping the version number
2. Ideally get sign-off from another reviewer/maintainer
3. Merge the PR
4. Run cargo publish from the main branch
5. Create a git tag for the commit

[RunDevelopment]

No worries! I requested the transfer.

Release workflow

👍 I'll make a 0.0.1 release after the transfer is done.

I'm still not happy with the API, so I made a milestone with issue of what still needs to be done.

[RunDevelopment]

I'm going to close this issue now that it has served its purpose.

DDS support will be moved to image-extras (#2565) and I already made a PR for it there (image-rs/image-extras#16).