The electron affinity for nitrogen is in fact a positive value (about +9 kJ/mole, so this is an endothermic process), so work must be done to force the electron onto the atom:
9 kJ/mole + N(g) + e- --> N-(g)
Since the electron affinities of most neutral atoms are negative, indicating exothermic processes and a "downhill" potential energy change, the positive value for nitrogen means that there is a much lower than expected attraction for the electron by the neutral nitrogen atom.
In fact, this is generally true for all of the Group 5A elements. While the electron affinities of the other elements in this group are negative, they are generally much less negative than the electron affinities of their neighbors in Groups 4A and 6A.
The explanation is simply that, with the Group 5A elements, the added electron must go into a p-orbital set that is already half-filled, a more stable arrangement. This stability is lost when another electron is added. Elements with filled (such as the Group 2A elements) or half-filled valence subshells generally have more positive, and therefore less favorable, electron affinities.
Some textbooks say that the reason adding an electron to nitrogen is unfavorable is because the added electron must go into a p-orbital that is already occupied by an electron. This makes sense when comparing the electron affinities of the Group 5A elements with Group 4A, where you have an empty p-orbital to put the extra electron in. But then the Group 6A elements should have the same problem, but in fact their electron affinities are in the more "normal" range, so electron-electron repulsion effects are less of a factor with these.
Hope this makes sense! Most chemistry textbooks, at the college level at any rate, make some mention of trends and abnormalities in ionization energies and electron affinities of the main group elements.
Steve