All work
Utility experience / Research leadership & Design Systems

Glass or familiar?
Let users shape the direction.

A/B concept testing with 34 participants to choose a design-system direction—combining visual preference with observed payment and navigation behaviour.

34Concept-study participants
2Design directions compared
8Designers guided jointly
2Customer channels
My role

Research Lead, jointly owning design direction with the Design Lead. I coordinated research, design and UX writing outcomes, reviewed work and connected findings with delivery decisions.

01 / Problem & decision

How far could the experience evolve without losing everyday clarity?

A utility app and website serving approximately seven million users in the project context needed a shared design direction. We had two alternatives: a contemporary glass-style interface, or an evolution closer to the existing visual language.

The starting point was not a product that users uniformly disliked. Participants said the existing experience supported their main needs: viewing and paying bills. The challenge was to understand whether a more substantial visual change would be accepted—and whether the new concepts still supported those familiar tasks.

Concept A / Transform

A glass-style direction

A more contemporary visual treatment with layered cards and a different presentation of account information. We wanted to understand acceptance, interpretation and the learning involved.

Concept B / Evolve

A more familiar direction

A design closer to the existing experience. We used it as the alternative for evaluating visual preference and the clarity of key interactions.

The problem statement

How might we select a design-system foundation that users want to adopt, while making essential bill-payment and account-management tasks clear enough to complete?

01

Acceptance

Which overall look and feel would users prefer, and why?

02

Usability

Could they pay, change account selections and adjust an amount?

03

Adaptability

How did they interpret usage information and respond to light and dark modes?

Project evidence: Concept Study, pp. 6, 15–22. Concept descriptions and product scale also reflect my project account.

02 / Research approach

Preference told us what appealed. Tasks showed us what needed work.

I led the research and jointly owned the design direction with the Design Lead. We combined contextual questions, moderated usability tasks and A/B concept evaluation so that a visual preference would not become our only decision criterion.

UnderstandCurrent bill habits
ObservePrototype tasks
CompareA/B preference & reasons
SynthesiseDirection + refinements

Why use A/B concept testing?

We had two concrete directions and needed users to experience them before choosing. The questionnaire asks participants to compare payment, usage displays and dark themes, then explain their overall preference.

This made the decision more specific than asking whether they liked a new design in isolation.

Why add usability tasks?

A preferred interface can still make a task harder. Observing actions helped us identify hesitation, interpretation errors and recovery, then decide which interactions needed to change within the selected direction.

The task block was planned for 45 minutes, allowing both structured activities and follow-up questions.

What “A/B testing” means in this case study

The project called this A/B testing: a moderated comparison of prototype concepts and preferences. It was not a live traffic experiment measuring conversion uplift. NN/g’s live A/B guidance describes a different statistical setup. Here, task percentages describe the tested sessions and help prioritise refinement.

NN/g reference [1]

Project evidence: Concept Study, pp. 6, 8, 23–27.

03 / Recruitment & participant profile

Recruit people who already manage the real task.

We recruited 34 participants. The inclusion criteria focused on recent product use, adult account responsibilities and the ability to engage with the prototype in a supported language.

Recent experience

Existing app or portal users who had used the platform within the previous three months.

Digital access

Aged 18 or above, with a smartphone or computer, and comfortable using digital services.

Language access

Able to read and understand Bahasa Melayu or English.

Recruitment priorities recorded in the study plan
GroupPlanned coverageWhy include them?
Regular users34 overallUnderstand navigation and discoverability in bill-viewing and payment journeys.
Older users8 aged 55+, using the app independentlyExplore hierarchy and whether screens could be understood without help.
Younger users8 aged 18–25, using the app independentlyExplore expectations of modernity and functionality.
Low-vision users5; colour-vision differences desirableInvestigate font sizing, contrast and zoom-related needs.

These are overlapping recruitment targets, not four additive groups. The report states that older-user recruitment was met and younger recruitment was nearly met; it does not provide a verified completed low-vision count. Concept Study, pp. 7, 10.

Where participants came from

Selangor
19
Kuala Lumpur
7
Terengganu
1
Johor
1
Melaka
1
Undisclosed
5

Reported counts total 34. Concept Study, p. 9.

Age profile

20s
6
30s
7
40s
10
50s
8
Not accounted for
3

The published age chart totals 31. The grey remainder represents the difference from n=34, not an inferred age group. Concept Study, p. 10.

What this sample helps us understand

IT was the largest occupational group (13), followed by telecommunications (4) and architecture (3). This gave us useful feedback from digitally comfortable users, but it also limits how confidently we can generalise to people with low digital confidence. Location was similarly concentrated in the central region.

Our initial expectation was that older users might prefer an incremental visual change. My recollection of the sessions was more nuanced: older participants also responded positively to the contemporary direction. The report does not publish an age-by-preference cross-tab, so I present that as a qualitative learning rather than a quantified age effect.

Project evidence: Concept Study, pp. 9–11; age-related interpretation from my project account.

04 / Questionnaire & task design

Ask about the context, then watch the behaviour.

The discussion guide connected existing habits with what participants saw and did in the prototypes. Questions about multiple homes were particularly useful: the person monitoring a bill was not always the person paying it.

Selected questions from the guide, condensed and anonymised
StageKey questions / promptsWhat we wanted to learn
Before the tasksHow many homes do you pay electricity bills for? Where do you view and pay? Which payment methods do you use?Account responsibilities, existing habits and channel choices.
First impressionsWhat do you see here? What do you think this icon does? What would happen if you tapped it?Understanding before exploration or explanation.
Account interpretationWhose usage is shown? What does a favourite bill mean? How many bills would this payment action include?Whether account scope and payment scope were clear.
A/B comparisonAfter using both designs, which payment experience do you prefer, and why? Which usage display is easier to understand?Experience-based preference, with reasons.
After the tasksWhat do you think of colour, text size and spacing? What would you change? Which design would you choose to use?Overall acceptance and priority improvements.
01

Pay the full bill

Show how you would pay. Explain what you understand when bills appear in the payment list.

02

Choose accounts & edit

Select the parents’ and rental-home accounts, then change the parents’ payment amount to RM100.

03

Change the theme

Switch from light to dark mode, return to the main screen and describe the experience.

We also explored login icons, registration and recovery entry points, account usage and chart comprehension. The guide introduced prototype limitations and included follow-up probes to understand participants’ reasoning.

Why the wording and sequence matter

Asking what a control means before explaining it helps reveal interpretation problems. A specific account-and-amount scenario tests more than the easiest payment path. Asking “why?” after preference makes the findings actionable: it helps distinguish visual appeal from information clarity. For a repeat study, I would also document and counterbalance concept exposure order; the report does not establish that procedure.

Project evidence: Concept Study, p. 8 and Appendix A, pp. 23–27. Rationale is my interpretation of the study design.

05 / Why 34 participants?

Size the study around the decision and the risk.

~7 millionProduct scale

The reach makes design decisions consequential.

34Research participants

A focused sample for concept direction and usability learning.

The study combined qualitative observations with descriptive task metrics. Its sampling plan referenced NN/g, recruited 34 participants and allowed for further recruitment if patterns had not formed. The report states that findings continued to repeat and that the team reached thematic saturation.

Qualitative reasoning

We needed to understand acceptance, confusion and recurring behaviours. NN/g recommends analysing interviews as research progresses and extending recruitment while important new themes are still emerging. Scope and participant diversity affect how much research is needed.

NN/g reference [3]

Quantitative reasoning

Sample planning depends on the metric, acceptable uncertainty and decision risk. The report cited a 28-participant benchmark; the assumptions behind that option are explained below. Task-level sample sizes still need to be considered separately.

NN/g reference [2]

An applied-research trade-off

In industry-led research, a 90% confidence level can be acceptable when the consequences of uncertainty are manageable. NN/g describes this as an option for practical UX research, with 28 participants needed for a binary metric at a ±15 percentage-point margin of error under its stated assumptions.

Our 34-participant study informed an iterative concept decision, supported by observed behaviour and repeating themes. This provides a rationale for the scale of the research; it does not establish that every reported task result achieves that confidence or precision.

Read NN/g’s sample-size assumptions [2]

For a large population, sample requirements are driven primarily by the metric, precision and risk assumptions—not a fixed percentage of all users. Industry research can accept a different precision–risk trade-off when the decision warrants it; it still needs those assumptions to be explicit. The task subsets in this report cannot inherit the confidence or precision of the full sample.

Scope of the evidence

The study supports a formative direction decision. Recruitment was concentrated geographically and occupationally, and some task denominators are not reported. A live A/B experiment to establish conversion uplift would require a separate sample calculation and experiment design. The report’s saturation statement concerns repeated themes, not statistical precision.

Project evidence: Concept Study, p. 6. NN/g links below explain the conditional sample-size guidance.

06 / Results & patterns

Concept A won on appeal. Concept B exposed a better payment interaction.

Overall look-and-feel preference

A clear direction within the study

76.48%Concept A · glass-style23.52%Concept B · familiar

Percentages reproduced as reported, including source rounding. Overall study n=34; the report does not provide raw preference counts. Concept Study, p. 22.

The overall preference supported Concept A as the visual foundation. But task performance showed where its interactions needed refinement. That distinction was the most valuable outcome of combining the two methods.

First-attempt task completion

Concept AConcept B

Pay full amount

A
100%
B
100%

Select accounts & edit amount

A
46.67%
B
76.47%

Switch to dark mode

A
100%
B
100%

Reported percentages, not population estimates. Per-concept task denominators are not explicitly stated, so they are not assumed to be 34. Concept Study, p. 15.

The more demanding payment task revealed the gap
OutcomeConcept AConcept B
Completed first attempt46.67%76.47%
Completed after multiple attempts46.67%23.53%
Could not complete6.67%No failures reported

Users struggled to work out how to change the amount and account selection from the payment review screen. Concept A was more appealing overall, but its lower first-attempt completion made payment refinement a priority. This was not a reason to discard the whole concept; it was a reason to separate the visual foundation from the interaction details.

Anonymised comparison of the existing home screen and Concepts A and B showing payment placement
Home-screen comparison corresponding to Concept Study, p. 19. Branding removed in an edited visual; fine interface details may differ. View larger
Observed friction

Seen did not always mean acted on.

In Concept A, participants noticed and understood the payment action but hesitated before selecting it. The report describes a clearer relationship between the amount and adjacent payment action in Concept B.

Design response

Keep the amount and action together.

Use Concept B’s payment placement to inform the selected direction. This directly addresses the observed hesitation, without claiming eye-tracking measurements or proven conversion uplift.

Principle tags explain the design rationale retrospectively; they are not additional study results or a separately scored heuristic audit.

Closer look · 01

Editing a payment needs to feel like an available action.

Account selection & amount editing · Concept Study, pp. 15, 26

Proposed interaction structure
Selected accountParents’ homeChange account ↗
Payment amountRM100.00Edit amount ↗
Review account + amount
Conceptual schematic of the recommendation; not a shipped screen or additional test result.
What happenedUsers struggled to change the account and amount from the review screen.

First-attempt completion was 46.67% for A and 76.47% for B. Some participants recovered after several attempts; A also recorded failures.

Why it matteredPaying a bill is not always paying every account in full.

The scenario deliberately combined two homes with a custom amount. It exposed friction that the successful full-payment task did not reveal.

Design responseMake editing explicit before confirmation.

Keep the selected account and amount visible, provide recognisable edit controls and let users review the changes before committing.

Closer look · 02

“Where am I?” came before “Where is the chart?”

Account usage discovery · Concept Study, pp. 20–21

Observed navigation pattern
01 / StartHome overviewChoose another account
02 / ArriveA similar-looking pageAccount context was not obvious
03 / FrictionRepeat the path or stopThe usage chart remained below
Reconstructed from the reported journey; no participant-level counts implied.
What happenedParticipants repeated the journey after reaching the account page.

Some stopped looking for the chart. After an explanation, they understood that the page resembled home and that the chart required scrolling.

InterpretationThe transition did not communicate a new account context clearly enough.

Visual consistency is useful, but making an overview and a detail page too similar can obscure their different purposes.

Design responseDifferentiate the detail view and surface its purpose.

Use a clear account heading, a visible route back and a recognisable usage section. Retain the bar-chart format participants found easier to understand.

The context behind “view, then pay elsewhere”

Account responsibilities

Single account
71%
Multiple accounts
29%

Reported account distribution, Concept Study, p. 12.

Viewing and paying are different jobs.

Some participants monitored accounts for relatives or business outlets while someone else made payment. Others paid individually because they needed a separate receipt for each transaction.

Favourite accounts could help distinguish bills they usually paid from accounts they only monitored.

Reported payment-method mix

Saved online-banking bill payment
28%
In-app bank transfer
50%
In-app card
19%
In-app wallet
3%

Rounded percentages from Concept Study, p. 13; response base is not specified. Methods are generalised to remove provider names.

Participants described established habits around saved banking payments, and some felt safer using their banking app. Better button visibility therefore addressed only part of the problem. Payment design also had to respect account ownership, receipt needs and channel trust.

Screen-level findings that shaped the next iteration
Screen / interactionWhat we learntDesign implication
OnboardingSome users skipped when allowed; others read to discover what had changed.Keep a clear skip route and make the value of the update easy to scan.
LoginThe facial-login icon was sometimes mistaken for a camera or scan action; an identity-service logo was unfamiliar.Pair unfamiliar authentication symbols with clear labels.
Account usageIn Concept A, a subpage looked too similar to the home screen. Users repeated the path or missed the chart below.Differentiate account detail from the home overview and make context visible.
Usage chartParticipants preferred a bar chart for understanding usage; familiarity with the existing display helped.Choose chart form for comprehension, not visual novelty.
Dark modeBoth concepts recorded 100% first-attempt switching in the reported task results.Carry theme support into component and token planning.

The overall pattern

People could welcome a contemporary visual direction while still needing familiar, explicit task cues. The strongest solution combined the preferred look and feel with clearer payment placement, account context and information hierarchy.

Project evidence: Concept Study, pp. 12–22. Design implications are synthesised from the findings; they are not all verified release outcomes.

07 / Final direction

Adopt the preferred foundation. Refine the parts that slowed people down.

Concept A

Preferred visual foundation

Interaction learnings

Clarity from Concept B and observed tasks

Design System direction

A coherent basis for continued refinement

I translated the study into a direction the team could work with: carry forward Concept A’s visual language while resolving the specific usability issues before treating the library as a stable foundation.

Visual foundation

Align the team on the contemporary direction and define how it behaves across light and dark themes.

Interaction priorities

Improve amount editing, account selection, payment prominence and the distinction between overview and detail.

Reusable standards

Develop component guidance, semantic token naming, responsive behaviour and complete interaction states with the Design Lead and developers.

The result was a direction decision and a refinement agenda, not a claim that every screen in the winning concept should ship unchanged. Subsequent discovery, SME reviews and usability validation extended the work into registration, authentication, account settings and professional-user journeys.

Direction evidence: Concept Study, p. 22. Subsequent process: MVP Research Report, pp. 6–8, 29–42. Design System responsibilities reflect my project role.

08 / A separate validation round

Core payment tasks worked. Discoverability still needed attention.

34Concept comparison
15Later prototype validation

Different study stages. They are not pooled into one sample or assumed to be unique participants.

The later draft report documents 15 participants and says findings were repeating. It includes different task subsets, so the overall study size is not the denominator for every task.

Reported individual task results

Payment and bill-viewing tasks

Link a card6 / 6
Pay via bank transfer6 / 6
Pay by card6 / 6
Pay a specific amount6 / 6
View PDF bill6 / 6

Each square represents one successful task attempt in the reported six-person subset. These counts describe prototype sessions, not live customer conversion.

Change password

  • 2 first-attempt successes
  • 2 completed after errors
  • 2 failures

Task location, instructions and recovery needed further attention.

Finding meter information

The draft identifies this as a major discoverability problem. The design implication was to review labels and the location of account information.

Successful payments did not remove the need to improve supporting journeys.

How the draft data is presented here

The report’s recruitment targets total 16, while its completed sample is 15. Some demographic totals and task summaries also differ. I use the explicitly documented six-person payment subset and the internally consistent 2/2/2 password breakdown, rather than derive a combined success rate from incompatible totals. Detailed findings were still marked as under review.

09 / Design System & delivery

Turn a research decision into something teams can use.

01

Build the foundation

Jointly set the Design System direction with the Design Lead; organise component libraries and semantic naming practices.

02

Make behaviour explicit

Review default, focus, error, loading and empty states, responsive behaviour, long content and accessibility considerations.

03

Support adoption

Document component usage and do/don’t examples, share a writing playbook, and guide designers and developers.

EvidenceObserved needs
StandardsComponents & states
HandoverImplementation guidance
ReviewDesign consistency

I reviewed semantic naming best practices with developers and worked with architects on feasibility. My coding background helped me translate interaction intent into implementation discussions.

Resource allocation under pressure

I coordinated two researchers and shifted designers between prototype preparation, usability support, rapid fixes and existing deliverables. An additional stakeholder exhibition required presentation materials, speakers and a live feedback setup.

Shared ownership with clear boundaries

I guided eight designers through reviews and feedback in a shared reporting arrangement. Project management maintained formal administration and final sign-off; I contributed requirements, sequencing and review decisions.

10 / Outcome & reflection

A defensible direction, with a clear next validation step.

What the work established

  • A research-informed foundation for app and web.
  • Concrete payment, navigation and registration improvements.
  • Shared component, content and state guidance.
  • Further validation beyond the initial concept decision.

What I would measure next

  • Live payment completion and abandonment.
  • Account-selection errors and receipt-related friction.
  • Independent task success across key segments.
  • Accessibility and component consistency in implementation.

The leadership lesson

My role was to help the team make a decision with evidence, make its limitations clear, and turn that decision into standards people could implement. The work demonstrates research-led delivery; the supplied evidence does not establish a measured production adoption uplift.

Evidence & methodology references

Trace the decisions back to their sources.

Project sources are cited by neutral labels and page number throughout. Source reports remain off the public website to avoid exposing client identities and participant information.

Concept Study

September 2025 · 28 pages. Primary evidence for the 34-participant A/B concept comparison, recruitment, demographics, questionnaire, task results and preference findings.

MVP Research Report

October 2025 · Later discovery and validation. Read alongside the later validation DOCX; its 15-participant study is kept separate from the original concept comparison.

  1. NN/g — A/B Testing 101

    Explains live experiment design; helps distinguish this prototype comparison from live conversion testing.

  2. NN/g — How Many Participants for Quantitative Usability Studies?

    Explains why sample recommendations depend on confidence and precision, including the conditional 28-participant option.

  3. NN/g — How Many Participants for a UX Interview?

    Explains saturation, recruitment relevance and analysing as research progresses.

  4. NN/g — 10 Usability Heuristics for User Interface Design

    Jakob Nielsen’s principles inform the tagged recommendations for control, prevention, recognition, status and consistency.

  5. NN/g — Proximity Principle in Visual Design

    Supports grouping the payment amount and its related action.

  6. NN/g — The Aesthetic–Usability Effect

    A lens for interpreting visual preference alongside task performance.