Elder abuse and neglect has become a global public health concern. Data and studies of elder abuse in Ghana is very scarce and no study exist of health workers perceptions of elder abuse in Ghana. Current sociocultural and economic trends have contributed to changes in the phenomenon of elder abuse requiring serious efforts to prevent detrimental effects of abuse and neglect of older adults. A semi-structured interview guide was used for data collection. Using content analysis, the findings showed that elder abuse and neglect occur in both hospitals and nursing homes, which might be attributed to different personal, situational and institutional characteristics as well as cultural and traditional value systems. Various factors at the level of interpersonal relationships contributed negatively and positively to elder abuse and neglect between residents themselves, and between the health workers themselves as well as between residents and the health workers. Some residents’ perception that they are socioeconomically higher than others seemed to be implicated in their frequent conflicts with other residents and with health workers. In other instances, when health workers perceive excessive power for being ‘a carer’ in their roles in providing care for ‘a dependent’ resident, this might be misused in a power relationship, which could either contribute to elder abuse or used in a way to provide adequate care. Situational characteristics such as aggressive exchanges between residents and health workers and institutional characteristics such as limited facilities and resources to care for residents are all factors that were implicated in elder abuse and neglect. Implications of findings are discussed within the frameworks of Bronfenbrenner’s Ecological Model and the Social Exchange Theory. More specifically, the findings also have implications for interactions between the microsystem level and the exosystem and how the exosystem such as social welfare services and mass media can play a role in the management of older adult abuse and neglect. Finally, culture and traditional views, beliefs system and the socioeconomic levels seem to be implicated in elder abuse and neglect.