Terrain coloring paints the terrain verts any color you choose, accessed with the number 3 key. It's mostly used for adding shadows, "baked-in" lighting, and other such details to maps. This feature has a lot to do with the look of the maps in AMRTS. It can add greatly to the depth and realism of a map, without effecting unit speed or line-of-sight the way textures and geometry do.
There are two colored circles in the panel, with the same design as choosing a team color in a multiplayer lobby. Click a circle to change its color. The top one represents the color that will be applied if you left click. The bottom, if you Lshift+left click. (You can set two different colors for convenience) Once you have your colors, left click the terrain and start dragging to draw with that color. Use Numpad +/- to change the brush size.
You can further tweak how the color is applied with Circle Gradient, Circle Sharp, Circle Negative, and Irregular brushes. The slider just below these brushes represents the "pressure" at which the color is applied.
You should find it helpful to load some of the singleplayer missions in the editor and see how the stock mission designers used terrain coloring.
There are some other miscellaneous controls tied to this panel:
If you are using the unpatched editor, you will see different controls which relate to water opacity, wave intensity, and clouds. The 2nd-to-last slider is for the skybox.