Future wireless communication systems face several transceiver hardware imperfections that may significantly degrade their performance. In-phase (I) and quadrature-phase (Q) imbalance (IQI), which causes self-interference effects on the desired signal, is an important and practical example to these impairments. In this paper, a channel state information-assisted dual-hop amplify-and-forward (AF) relaying system in the presence of IQI is analyzed. The error performance of the relevant AF cooperative protocol is firstly studied by considering the traditional maximum likelihood detection (MLD) algorithm as a benchmark. Then, two compensation methods, weighting and zero-forcing, are proposed to mitigate the IQI effects. Finally, an optimal MLD solution is introduced by adapting the traditional MLD technique in compliance with the asymmetric characteristics of the IQI. The system performance is evaluated in terms of average symbol error probability (ASEP) through the computer simulations. The ASEP is calculated analytically for the optimal MLD method as well under the assumption of point-to-point communication, which has been envisioned as an allied technology of the fifth generation (5G) wireless systems, between the source and the relay nodes. A power allocation algorithm is provided for this specific case. The extensive computer simulations and analytical results prove that the proposed optimal MLD method provides the best results.