Jump to content

old JPEG, new format extensions...


Recommended Posts

well, yes, there is always good old JPEG, and it is fairly standard.

but, it has limitations, like for example, that in its standard form it lacks things like an alpha channel, and is always lossy (no "good" lossless mode).

this is mostly beng posted here in the off chance that it may be interesting or relevant.

(note: all this assumes knowing how a JPEG works internally).

several projects have added extensions, for example, some of my own extensions are a little more ambitious, and described here:


many of these extensions can be used without breaking compatibility with existing decoders (yes, I have tested).

these are mostly used in my case for Motion-JPEG based video-mapped textures (initially motivated by the relative pain it would be to add side-data in an AVI container), but may also be applied to plain textures (as an alternative to storing the texturemap in multiple files). this is used in the context of a 3D game-engine style project (usually with the files being specially built for a special purpose, rather than being intended for general purpose file interchange).

most relevant (in my case) is probably one of my older extensions, namely, an alpha channel:

an APP11 marker is used to identify a "component layer", namely "Alpha", and a sequence of APP12 markers with a tag of "LDAT" can encode an alpha sub-image, which is basically just a monochrome JPEG image.

(the idea here is basically that of the RGB being encoded in the basic image, with another contained mono-image giving the alpha-channel contents).

several other libraries (such as libjpeg, at least a newer / in-development version) have added alpha transparency in other ways (I forget the specifics, but IIRC it was a bitmap encoded via an APP9 marker or similar). (I am not aware of a spec for the libjpeg extensions).

the exact mechanism used isn't particularly important, but JPEG+alpha is a desirable feature IMO.

(even if not blessed by any particular standard, and one of those features commonly-accepted to not exist...).

also both my extensions and libjpeg have added support for a lossless mode via the use of the "SERMS RDCT" transform, though there are some minor differences. mine simply uses a different colorspace transformation, known as ORCT or RCT (this colorspace was also used in JPEG 2000), at the cost that colors are "slightly off" with an unaware decoder, whereas libjpeg is using a "residual encoding" scheme. (side note: lossless mode also uses 4:4:4, or IOW, there is no chroma subsampling).

or, stated more simply: a lossless mode which is mostly compatible with the normal lossy JPEG.

in my own tests, I have seen slightly better compression than PNG while going this route, although at the cost of worse decoding times. however, given that RDCT and DCT are value-compatible, it is still possibly to more quickly decode the image (with some loss) by using the normal/faster DCT transform (RDCT is basically DCT, but is able to inverted exactly).

here is a paper about the topic (part of where I got the idea from):


not sure if any of this is at all interesting / relevant here...

Link to comment
Share on other sites

Hi cr88192, - as you said - I doubt that you will find in this forum a lot of people having an idea about the internals of jpeg.

The jpeg format has so many drawbacks that it doesn't make big sense to extend it.

Maybe at some time Googles WebP will replace it. Because Google has some power to introduce a new format.

midoras signature.gif

Link to comment
Share on other sites

JPEG is an old format, but isn't really all that bad.

it does have a few merits though:

it supports lossy compression, which can produce smaller files than PNG;

it is not covered by existing patents (they expired, 1), unlike HDPhoto / JPEG-XR;

it is more widely supported than formats like JPEG2000 or JPEG-XR (which are not compatible with the older JPEG, and internally very different);

it is also "not all that complicated" (it is mostly just that T.81 is verbose and filled with rarely-used features);


1: patents are undesirable, as they allow "spooky legal action at a distance". MS says they wont sue people for using HDPhoto / JPEG-XR, but it is much less certain if this applies to modified/extended form of the format. but, the older JPEG is free of patents, because pretty much all of them expired.

given the extended form is mostly compatible, JPEG benefits from at least some compatibility with existing software.

for example, I can feed images in my format into Paint.NET (and other tools), just with the content of any extended layers being discarded (only the RGB components of the base-image are visible, everything else is ignored).

likewise, if the image was using RCT / ORCT, the colors are off (the image is tinted and over-saturated), so generally I default to the YCbCr colorspace (except in lossless mode).

these are generally acceptable downsides, for the most part, although I would sort of like to see alpha-channel support in PDN, but understandably, this much is unlikely.

(as-is, I mostly end up having to use PNG files here...).

(as-is, using the full capabilities of the format generally involves packing to/from a pile of PNG images though).

granted, there is also the option of me going and implementing JPEG-XR support or similar.

oh yeah, nevermind the video-codec related extensions.

kind of a similar issue, mostly just replace JPEG-XR with MPEG-4...

(hacked MJPEG vs MPEG4, but OTOH, for animated textures it doesn't matter as much if the size/quality tradeoff kind-of

sucks...). yeah, there is also Theora, but I would sort of need to hack it in a few areas anyways for the domain (IOW: for an alpha channel and multi-layer video).

note: strict dependence on OS-specific codec APIs is something to be avoided (though they are supported by the engine), but these APIs don't support these sorts of features either. so, most of the video decoding/playback is custom code (shimmed together JPEG decoder internals and OpenGL calls...).

or such...

Link to comment
Share on other sites

Then spent the time and write your own filetype plugin to support your format.

yeah, I could do this, maybe.

granted, would have to go look into the specifics of PDN plugin writing for this, I may need to look into this (and see if there are any specs/tutorials around for this).

thus far I have mostly been using PNG as an intermediate format (and converting files into my formats mostly from collections of PNGs), and also mostly working on my game project (trying to make it not suck...).

or such...

Link to comment
Share on other sites

To be honest, WebP sounds like it will (in time) be the death of JPG. Compression is quoted as being better than JPG, and it has the might of Google behind the new format.

The trouble with extending the existing JPG format is that other applications will not support the enhancements. Portability issues?

Link to comment
Share on other sites

To be honest, WebP sounds like it will (in time) be the death of JPG. Compression is quoted as being better than JPG, and it has the might of Google behind the new format.

The trouble with extending the existing JPG format is that other applications will not support the enhancements. Portability issues?

well, WebP, JPEG-2000, and JPEG-XR, are all basically in the same boat, in that each is trying to replace the original JPEG.

the main difficulty though is that JPEG is fairly solidly entrenched, and it is difficult to do *considerably* better than JPEG.

each also has big supporters. like, JPEG-XR is basically just Microsoft's HD Photo format, and has MS backing it.

usually, they add a few features, and may slightly improve the size/quality tradeoff, not usually significantly.

as for extending JPEG, mostly it becomes a matter of being selective as to where and how the extensions are used.

like, you can't expect a program that doesn't know about alpha channels to be able to decode them.

(so, areas that would be transparent just come out as big black areas or similar).

(the most conservative option though is just not to use any extensions, then there are no issues).

but, OTOH, if your app doesn't know about a format like JPEG-XR or WebP, essentially, it wont be able to open the file at all.

big difference?... (would you rather have a file that doesn't open entirely correctly, or one that doesn't open at all?...).

it can be made to look like a different file-format easily enough by using a different file extension (say, to indicate to users, "hey, this file is special..."). (in my case this basically amounts to using ".btjp" and ".btmv" in place of ".jpg" and ".avi").

(this is also partly relevant for my 3D engine, which due to current architecture, will only load single-layer RGB or RGBA if using a ".jpg" file extension, and can't directly use raw ".avi" files as a texture without a shader-file, so yeah, it is a little stupid here...).

otherwise: now, as for writing a plug-in for Paint.NET, I started messing with it earlier, but in my case it involves the relative mess that is porting a bunch of C code to C++/CLI and trying to glue it onto C# (and not being particularly high on motivation). mostly did the first part already, but admittedly am not strongly familiar with the .NET APIs involved (like how to convert between Bitmap and raw byte arrays, ...). (although a cleaner implementation would probably be to have a pure-C# JPEG codec, off-hand, I don't have one available).

or such...

Link to comment
Share on other sites

Why not use DDS for your 3D engine?

Link to comment
Share on other sites

Why not use DDS for your 3D engine?

because I am not using DirectX, and DDS offers no other major advantages to justify its drawbacks...

DDS uses DXTn, which is basically a fixed-rate lossy encoder (basically, it packs 4x4 pixels into 64 or 128-bits).

although it stores textures in less space than a raw BMP or TGA typically would (making it an attractive option vs BMP or TGA), it will generally produce a considerably larger image than a PNG or JPEG would.

the main advantage DXTn has is that it is supported natively by the graphics hardware, and can be loaded faster (basically, fed more directly to the hardware). this is in contrast to normal texture loading, which will first decode the JPEG or PNG, then porentially resample it for mipmaps, and then potentially DXT compress it.

however, many of us would rather have the textures in a more compact representation on disk, and for things like streaming video to a texture, there is the option of basically using raw/uncompressed RGBA and not using mip-maps (this makes video-mapping cheaper, with the drawback of video-mapped textures being a little more prone to alias).

side note, assuming some sort of Motion-DDS:

a 256x256 texture, with mipmaps, would take roughly 128kB per frame (DXT2/3 + mipmap);

at 10Hz, this basically means about 1.28MB/s (10.24Mbps ).

by video-codec standards, this would be, sort-of, dismally bad...


in contrast, doing similar with raw RGBA + mipmaps, would take 512kB/frame, or 5.12MB/s, or 42 Mbps.

(this is part of why uncompressed RGBA video is just scary, and why capturing it can quickly eat up ones' HDD...).

edit again:

basically (technology wise), DXTn is to JPEG, like what ADPCM is to MP3.

your ADPCM-WAV will be smaller than your raw-PCM WAV, but will be considerably larger (and probably also sound somewhat worse) than your MP3.

Edited by cr88192
Link to comment
Share on other sites

add, more concrete example (image size=512x512, a simple picture of a mountain with a transparent sky):

TGA, no RLE, 1.0MB;

TGA, RLE, 212kB;

DDS, full options (DXT5 + mipmap), 342kB;

DDS, (DXT5 + no-mipmap), 258kB;

PNG (saved by PDN, *1), 145kB;

PNG (saved by GIMP), 85kB;

GIF, 34kB (but looks awful);

JPEG (100%), 30kB.

JPEG (90%), 16kB.

*1: idle, request PDN fix whatever has gone terribly wrong with saving PNG files (broken deflate or poor compression level?...).

or, IOW, the DDS file is losing to the TGA in this case WRT compression...

and the JPEG is doing "pretty good" here.

so, yes, there are reasons I like JPEG...

Edited by cr88192
Link to comment
Share on other sites

The optiPNG filetype plugin allows you to optimize png images so they don't take up so much space.

No, Paint.NET is not spyware...but, installing it is an IQ test. ~BoltBait

Blend modes are like the filling in your sandwich. It's the filling that can change your experience of the sandwich. ~Ego Eram Reputo

Link to comment
Share on other sites


the PNG encoder in my 3D engine usually generates smaller images than PDN as well.

for example, I have a 460kB PNG screenshot (produced by my 3D engine), and PDN re-saves it as 721kB.

testing, GIMP will re-encode the image at 415kB.

I am not doing anything fancy in the PNG encoder, nor is my deflater particularly fancy either (this part is also custom written though, but it is still basically just using a greedy search).

I am left to suspect there may be an issue with the PNG encoder in PDN, like either it is making a bad choice as to the filter, or is compressing with a very low compression level or similar (1 or 2 or equivalent), or the deflater is broken...

so, a possible general feature request could be "fix whatever is wrong with the PNG encoder".

Edited by cr88192
Link to comment
Share on other sites

an experimental PDN plugin has been written and is available here:


it comes with source-code as well...

(I will for now give permission for people to "do whatever with it").

basically, after a failed attempt to write a plugin using C++/CLI (couldn't get Paint.NET to recognize it), I ended up basically doing a hack-job rewrite of a lot of the encoder/decoder code into C# (the code was originally written in C, and was more of a quick-and-dirty rewrite, no never-mind if it isn't exactly "good style").

currently, a lot of stuff doesn't work, but it would require figuring out how to work with loading/saving layers in PDN.

I make no claims about if/how-well it works, only that it seems to be working on my end...

currently, things like the quality-level and similar are also hard-coded (would need to make some sort of GUI or something for choosing options).

Edited by cr88192
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Create New...