GdkPixbuf afaik only supports 8 bits per colour.
If you want other formats, e.g. a 5/6/5 then this looks like it isn't possible in gdk and you would have to delve to lower levels.
In cairo; however, there is a lot more supported formats (in fact you won't actually have a true 24bit packing).
Set up an image surface as CAIRO_FORMAT_RGB16_565 if you want 16 bit colour (haven't seen anything for 8 bit) though you could make a hack with A8 possibly.http://cairographics.org/manual/cairo-Image-Surfaces.html
Oh and you may want to look into saving the window as a vector graphic (it can often be a much smaller size when in svg)