The shellbox program is a "chat"-like computer program that allowed Alex and Nikita to communicate whilst Alex is training at Division. It is encrypted so that it can not be traced, tracked or dechiphered anyone not meant to read the messages passed.
In "All The Way", it is shown that the shellbox program is something Birkhoff teaches to Division recruits. This leads suspect to fall on Nikita, and for the higher-ranking agents to believe that there is a mole amongst the ranks of Division. It is also shown that shellbox is something that Nikita created in her time as a recruit in Division, implying she had planned her escape for a long time.
Aside from Nikita and Alex, it is used by Percy when it is discovered in the eleventh episode. He poses as Alex to send a message to Nikita, whom is waiting for the newly-activated Alex to appear at the site of her set task. The program is destroyed by Birkhoff when Nikita re-escapes Division and Thom is labelled as the mole.