To trigger a popup message >> select Notify >> Notification Type: Popup as one of actions in your Custom Action or Workflow Rule.
Please find the example below. This Workflow Rule (created on a User Entity) triggers a popup message upon user Login.
For further clarification, please join our free Advanced Q&A webinars (to register >> http://www.clarizen.com/resources/online-demo.html#second ). One of our experts will be able to discuss your requirements and demo it for you accordingly.