Taboo slang for the act of masturbation, stimulation of the genital organs of oneself to achieve sexual pleasure.

This is the dictionary definition but the word is used metaphorically in the novel Cryptonomicon as can be read in this quotation from page fifteen.

"Okay. Meaning that math was more than a physics of bottlecaps."
"It appeared that way, Lawrence, but this raised the question of was mathematics really true or was it just a game played with symbols? In other words - are we discovering Truth, or just wanking?"

Wanking being a metaphor for academic self-gratification, an activity that is in the end pointless and without higher purpose. The question is when math stops relating to the physical world does it remain a truthful activity or is it moving symbols about according to arbitrary rules to appear to be truth.