As your dealing with numeric data you would most likely want spin buttons rather than entries.
Pack all nine of them into a GtkTable (see the tutorial at http://zetcode.com/tutorials/gtktutorial/gtklayoutmanagement/
If you use the same GtkAdjustment for the diagonally adjacent buttons then updating one will automatically update the other.
What you suggest about adding a new matrix when the user finishes can have things go wrong if the user enters something incorrectly and wants to change it later. It would be good to have another spinner that sets the number of matrices or an add/delete button (as well as potentially the option of reshuffling them in the queue if ordering matters). Gtk has some flexible data types such as GArray and GSList that can expand arbitrarily after their defined. Pack in the adjustments and you can retrieve the data later and reuse the variable names.
Matrices can occupy a lot of screen space and once they're entered they probably don't need to be shown in full. If you stick them in a GtkExpander then the user can access them any time and hide them when done.