Hang on are you still using gdk contexts? I thought they were depreciated.
I've always done this within the draw function; having a draw and a redraw function. The drawing area is set up as a custom widget with GObject (there's a tutorial on how to do this) that way I can set up signal functions with the mouse behaviour. You can set up some structures that will include the coords of the lines and text. You can then do a redraw when you get a mouse release signal etc. There should be no need to minimise the window.
I've got an example for a plotting widget (though its still in gtk2 but will be an easy transfer)
https://github.com/pchilds/GtkPlot -- the plotlinear one is the only one that is complete yet.