libserial.so is in package "pluto-libserial", there is no such package "pluto-serial-device", only "pluto-generic-serial-device" but it's completly different thing from CM11 device.
I'm not quite sure what happend on your system (I repeat CM11A is something VERY different from Generic_Serial_Device). So please remove the device you tried to add, then go to "Wizard" -> "Devices" -> "Interfaces" and add a CM11A device from there.
If you want you can check if cm11a and libserial are installed
dpkg -l 'pluto-cm11a'
dpkg -l 'pluto-libserial'
But I'm pretty sure that if you follow the steps everything should be fine.
Set the port which is used by CM11A, add some lights, quick reload the router and that's it.
Now regarding your communication error : WHAT is not able to comunicate with WHAT?
Is the message like :
Cannot communicate with router
If so, there is a problem with your router, make sure it's running, and the device you are trying ro run is same version with your DCERouter.
This shouldn't happend but if you upgraded some package by hand and not the whole pluto system this is quite possible.