An effective-field theory with correlations is developed for a mixed spin-1 and spin-3/2 Ising system with two alternative layers of a honeycomb lattice. Spin-1 atoms and spin-3/2 atoms are distributed in alternative layers of a honeycomb lattice. We consider that the nearest-neighbor spins of each layer are coupled ferromagnetically and the interaction between the vertically aligned spins and adjacent spins are coupled either ferromagnetically or antiferromagnetically depending on the sign of the bilinear exchange interactions. We investigate the temperature dependence of the total magnetization to find the compensation points and to determine the type of compensation behavior. We present the phase diagrams in different planes for h = 0, and the phase diagrams contain the paramagnetic, nonmagnetic and ferrimagnetic phases. The system also presents a tricritical behavior besides multicritical point (A), isolated critical point (C) and double critical end point (B) depending on the interaction parameters. (C) 2010 Elsevier Ltd. All rights reserved.