drivers/perf: arm_pmu: only use common attr_groups

Now that the 32-bit and 64-bit perf backends use the common groups
directly, remove the fallback and no longer allow the groups array to be
overridden.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
index 1a39899..60c065e 100644
--- a/drivers/perf/arm_pmu.c
+++ b/drivers/perf/arm_pmu.c
@@ -549,6 +549,7 @@
 		.stop		= armpmu_stop,
 		.read		= armpmu_read,
 		.filter_match	= armpmu_filter_match,
+		.attr_groups	= armpmu->attr_groups,
 	};
 }
 
@@ -1037,8 +1038,6 @@
 		goto out_free;
 	}
 
-	if (!pmu->pmu.attr_groups)
-		pmu->pmu.attr_groups = pmu->attr_groups;
 
 	ret = cpu_pmu_init(pmu);
 	if (ret)