Bringing the home in the lab: consumer-relevant testing for household electrical products

Abstract

Product testing is widely used to assess the characteristics, e.g. performance and energy consumption of a product. The procedures for executing the tests, including measurements and processing of measurement results, can be contained in standards. Standards should produce results that are repeatable, reproducible and valid at a reasonable cost. A number of stakeholders have questioned the appropriateness of several standards because the results that these standards provide are different from what a consumer may experience in practice. In the end, this can have negative consequences for the trust of consumers in the policy instruments (energy labels, minimum efficiency requirements) that use these standards and an energy savings deficit compared to what was expected by policy-makers. There is, therefore, a need for standards that better reflect ‘real-life’ conditions, meaning those conditions that consumers experience at home. However, unlike the other criteria that standards should meet, there is no methodology to assess the representativeness of a standard. This article develops such a methodology and presents the results for several household electrical appliances: washing machines, refrigerators and vacuum cleaners. In general, variation of situational conditions is not reflected in the standards and important parts of use behaviour are implemented as ‘artificial’ in the standards, showing that the other criteria (repeatability, reproducibility and cost) are prioritised over representativeness. Assessment of performance is difficult to evaluate because the type of assessment differs between practice and test; additional research is needed in this area.

Annex

Assessment of consumer-relevance of three example standards

Example 1: washing machines

Steps 1a, 2a and 3: Listing parameters, selecting (most) relevant parameters, assessment of correspondence

Parameters	Relevance for energy consumption (E) or performance (P)a		Range; average at consumerb (sources: preparatory study, (JRC 2016) EN 60160)	Setting/variation in standard (Source: EN 60456, IEC 60734)	Assessment of correspondence	Explanation of differences between standard and practice
Situational conditions
Temperature	Medium	E	> 0 to 40 °C; average	23 ± 2 °C	Average corresponds; variation not	Variation in standard is smaller than practice to respect working conditions and air conditioning costs in the lab
Humidity	Low		0 to ~ 100% r.h.; average	Not specified	No assessment possible
Inputs
Voltage	Medium	E, P	207–253 V; 230 V	230 V ± 1%	Average corresponds; variation not	Rated voltage in Europe
Frequency	Low		49–51 Hz; 50 Hz	50 Hz ± 1%	Average and variation correspond	Rated frequency in Europe
Harmonic distortion	Low	E	Distorted	Plane sinus	No correspondence
Water hardness	High	P	0 to > 10 mmol/l; average	2,5 mmol/l	Average corresponds; variation not	Performance assessment at different water hardness is subject of detergent manufacturer.
Water composition (e.g. metal content)	Medium	P	Potable water	Not specified. Water has to fulfil the water hardness specification. If total water hardness needs to be adjusted, it shall be prepared according to IEC 60734.	No assessment possible	Performance assessment at different water composition is subject of detergent manufacturer.
Water temperature	High	E, P	4 to ~ 25 °C; 15 °C	15 ± 2 °C	Average corresponds; variation not	Global average
User behaviour
Washing programme choice (Depends on type, soiling and load)	High	E, P	Cotton (60% choice), mixed, synthetic, wool + temperature	Standard allows a wide range of programmes.	Standard corresponds with practice, by offering capture of various programmes.	Regulatory provisions specify the programmes to be tested.
Type of cloths	High	E,P	Cotton, cotton—synthetic blends (20:80; 35:65; 50:50; etc.), wide range of synthetic fibres (PE, PAS, PP); wool, cellulose, viscose	(Standard-) cotton; standard also allows for other fabrics than cotton to be tested	Standard corresponds with practice, by offering capture of various fabrics.	Regulatory provisions specify the fabrics to be tested.
Soiling of cloths - quality - quantity	High	P	Natural soiling from food, body soils, ambient soil. Type of soils: soluble and insoluble soils, particular soil, colour soil A typical 3-kg Western European laundry load contains an average of 40 g of soil. (Pearce et al. 2005)	Artificial soiling of: carbon/oil, cacao, sebum, red wine, and blood, with as little variation as possible. One soil strip used per kg of loading capacity. One strip contains about 2.5 g of dried-on soils. Note that standard soil is more challenging to remove than natural soiling.	Correspondence on type of soil, not on quantity	High soil level needed to prove the machine is able to extract soil even at those conditions. Adherent soil needed to allow detecting measurable differences. Artificial soiling in standard for reproducibility and cost reasons
Load	High	E, P	< 1 kg to maximum of drum capacity (11 kg); average 3 kg	Standard allows testing from 1 to 15 kg	Standard corresponds with practice.	Regulatory provisions specify the loads to be tested: maximum (rated) capacity and its 50%.
Detergent type and amount	High	P	Powder, liquid, gel, or tablet; heavy duty/light duty; various compositions depending on brand/time. Amount according to detergent manufacturer or fixed amount or relative to load/soiling. Consumer research in Germany shows average amount of detergent dose of about 70 g (Kruschwitz et al. 2014) for an average load of 3.3 kg	Heavy duty powder detergent of fixed composition and dosed following formula depending on load size. Formula to calculate: 0 + 12 g/kg load. Fixed amount (40 g) needed for an empty drum. 12 g is added per kg of textile load (and soil strip) to achieve about an equal concentration for all load sizes, as more water is taken by the machine the more load is washed.	Average amount corresponds with practice. Differences in composition not. Differences in type (liquid) not.	Artificial detergent in standard for reproducibility reasons. Market detergents are changing continuously with time.
Product features
Capacity	High	E, P	3 to 11 kg; average 6.5 kg.	Manufacturer must show that the machine allows to clean clothes at highest level of filling which can be assumed to happen (with those clothes used for testing)	Standard corresponds with practice.	Maximum capacity captured by regulatory provisions.

1. Average: EU average (unless mentioned otherwise)
2. ^aCategorical assessment: low, medium, high
3. ^bRange: minimum-maximum (without extreme values)

Steps 1b and 2b: Listing of all performance aspects, selecting (most) relevant aspects, assessment of correspondence

Aspects of performance	Remarks	Assessment in practice	Assessment in standard
Removal of stains	Wide range of stains. Not all stains get always removed	Visual inspection of stains	Measured with reflectometer
Greying	Multi cycle effect	Visual inspection	Not assessed
Incrustation	Depending on washing conditions	Hardening of the textile	Not assessed
Textile strength loss	Multi cycle effect	Damage of textiles	Not assessed
Gentleness of action?	Multi cycle effect	Damage of textiles	Not assessed
Pilling effect	Multi cycle effect	Visual inspection	Not assessed
Colour damage	Depending on dye	Visual inspection	Not assessed
Dye transfer	Depending on dye	Visual inspection	Not assessed
Rinsing of soluble parts	Depending on detergent used	feeling	Alkalinity (EN 60456)
Rinsing of insoluble parts		Particles remaining	Not assessed
Rinsing of surface active ingredients		Feeling soapy	LAS rinsing (t.b.d.)
Rinsing performance—foam bubbles at end of cycle		Visual inspection	Not assessed
Wool shrinkage		Visual inspection	Measurable acc. to EN 60456
Germ reduction		infection	Not assessed
Spinning efficiency		Tactile inspection	Remaining moisture content (RMC) at max spin speed
Water consumption		Not assessed	measured
Energy consumption		Not assessed	measured
Duration of programme		Experienced	measured
Noise of washing		Experienced	measured
Noise of spinning		Experienced	measured
Annoyance of noise?		Experienced	Not measured

Step 3: Correspondence of assessment of performance

A number of aspects of performance are not assessed in the standard. For the aspects that are assessed in the standard, this is done through measurement instruments and not by human senses (seeing, feeling, smelling, hearing) as it is in practice.

Example 2 Household refrigerators

Steps 1a, 2a and 3: Listing parameters, selecting (most) relevant parameters, assessment of correspondence

Parameters	Relevance for energy consumption (E) or performance (P)a		Range; average at consumerb (source: VHK (2016))	Setting/variation in standard (source: EN 62552: 2013, impact of IEC62552:2015)	Assessment of correspondence	Explanation of differences between standard and practice
Situational conditions
Temperature	High	E, P	> 0 °C to 40 °C; 20 °C	16 and 32 °C.	Average to be determined; decision yet to be taken by the commission on which ambient temperature to adopt as that for the EU test	Via interpolation any value between 16 and 32 °C can be assessed. Likely that higher than usually encountered will be used for test to compensate for no door openings, no heat loading (food).
Humidity	Low (if no door openings)	E, P		< 75% r.h.		Humidity is of limited relevance in test if doors closed. But it is of higher significance in real use where doors are opened regularly. Hence, typical EU ambient humidity should be considered.
Air flow	Low (if no door openings)	E	No evidence identified on what is a typical air speed in EU kitchens	Air flow ≤ 0.25 m/s		Zero/low speed air is expected in a home most of the time. Also, the condenser is close against a partition and so the air flow is of little direct impact on condenser effectiveness. Air flow would have far more impact if door is opened.
Placing of the appliance during use/test	Medium		Refrigerators often pressed close against wall and between cabinets. Some are ‘built-in’ type. No evidence identified on what is the most common arrangement in EU homes	Placing against partition at rear unless specified (max. 50 mm from partition and 300 mm from sides, no partition above). Built-in appliances enclosed per manufacturer instructions.	Correspondence is reasonable for rear position, although most fridges in real use are pressed in from side without 300 mm gap to any partition.	Close-fitting side panels would impact thermal performance of appliance walls and reduce reproducibility. This could explain 300 mm gap introduced.
Inputs
Voltage	Low	E	207–253 V	230 V ± 1%	Average corresponds; variation not	Rated voltage in Europe
Frequency	Low	E	49–51 Hz	50 Hz ± 1%	Average and variation correspond	Rated frequency in Europe
Harmonic distortion	Low	E	Distorted	Plane sinus	No correspondence
User behaviour
Load: - Type (heat capacity) - Temperature - Amount	High	E, P	Various liquid and solid foods, input at various temperatures (5 to 40 °C)	No inserted heat load for EU regulation tests.	Appliance always has food or other loads in reality, but test for EU regulations is with appliance empty. Correspondence is poor, but compensation made via higher ambient test temperature	(Warm) load insertion under normal usage for energy label purposes (the energy efficiency test) is emulated by using higher ambient temperature than EU typical)
Door openings	Low	E	20 per day.	Currently not, but possible future simulation option via ‘load processing efficiency test’: loading with bottles of ambient temperature water (in one opening operation)		Door openings are emulated by higher ambient temperature (around 25 0C) and effect of door openings is small.
Defrost interval as a result of user behaviour and ambient conditions	Medium	E, P	No evidence sources identified so far.		Lack of door openings may not allow for consideration of frost thickness, defrost intervals and realistic energy consumption of defrost function.
Thermostat setting	High	E, P		Target average air temperature of compartments specified; thermostat settings such that target temperatures achieved
Arrangement of drawers/shelves	Medium	E, P		Instructions to maximise volume of colder space	Average corresponds; variation not	Arrangement in standard to measure at max conditions.
Product features
Internal volume (capacity)	High	E, P	Difference between volume as measured by standard and volume experienced as ‘available’ to user for storing food: presence or not of shelves etc. for the measurement is a key element.	‘Storage volume’ (‘usable’) current basis of calculation of equivalent volume used for EU regulation tests. However, IEC 62552: 2015 may no longer include ‘storage volume’, but ‘total internal volume’ (closer to ‘gross volume’)	Usable capacities substantially less than tested. Possible new ‘total internal volume’ substantially larger than ‘storage volume’ and over-statement would be even more exacerbated.	New IEC approach of total internal volume aims to simplify and improve repeatability, but in the expense of correspondence
Climatic class	Medium	E, P	Difference between the average/typical EU climatic conditions, and the functionality (allowed to be) declared by the supplier for each test	SN, N, ST, T	Nearly 3/4 of fridges claim tropical performance capability (> 38 °C), temperature rarely existing in EU. Hence, there is poor correspondence with EU situation.	Compensation factor for T and ST class in regulation makes attractive to claim tropical (and sub-tropical) allowance. Fridges suitable for tropical usage get allowance of 20% higher equivalent volume in regulation; sub-tropical of 10%
Built-in	High	E	Set-up per manufacturer’s instructions?	Specification per manufacturer’s instructions	OK
Frost free (defrost)	High	E	Market data: 40% has frost free function	Standard allows for measuring energy consumption of defrost.
Ice maker	Low (depend on ice production)	E		Measuring energy consumption of ice-maker when ‘ready for use’, but no ice production
Anti-condensation heaters				IEC 2015: If always on in normal use, then on for test; if user-controlled, test at max and min.
Other features, e.g. touchscreen/ internet			To be determined

1. Average: EU average (unless mentioned otherwise)
2. ^aCategorical assessment: low, medium, high
3. ^bRange: minimum-maximum (without extreme values)

Steps 1b and 2b: Listing of all performance aspects, selecting (most) relevant aspects, assessment of correspondence

Aspects of performance	Remarks	Assessment in practice	Assessment in standard
Cooling performance		Ability to cool food within an hour or two and keep it cool despite door openings	Cooling capacity test covers this well. Variation in temperature measured
Freezing performance		Ability to freeze several items of food at once and keep all frozen over time	Freezing capacity test covers this well. Variation in temperature measured
Freshness of food		Visual assessment	None at present.
Size of storage space	Metric for measured storage volume, as available over shelves for foodstuff storage no longer in IEC and EN test	As experienced by the when filling the space	Was measured as ‘storage volume’ but no longer in IEC. The new metric of total internal volume is much larger and less useful to user.
Food storage duration		Sensory assessment	Not assessed
Energy consumption		Not assessed (although indicated by proportion of time for which compressor is heard running)	Measured (with separate for defrost, ice-maker and anti-condensation but these do not appear on label)
Noise		Experienced	Measured

Step 3: Correspondence of assessment of performance

Several important aspects of parameters, especially regarding user behaviour such as (warm) load insertion and door openings, are emulated in the standard by means of a higher ambient temperature. Repeatability and reproducibility have been given priority over mimicking user behaviour in practice. Calculations show that the chosen interpolated ambient temperature represent the worst-case situation; this assumption needs to be checked at further revisions. The main gaps in assessing the performance relate to functional aspects, e.g. freshness of food and food storage duration, for which neither a metric nor a test method is available yet. The energy service aspects of performance, providing cooling and freezing capacity, are well captured.

Example 3: vacuum cleaners

Steps 1a, 2a and 3: Listing parameters, selecting (most) relevant parameters, assessment of correspondence

Parameters	Relevance for energy consumption (E) or performance (P)a		Range; average at consumerb (source: AEA (2009))	Setting/variation in standard (source: EN60312)	Assessment of correspondence	Explanation of differences between standard and practice
Situational conditions
Temperature	Low	E	> 0 °C to 40 °C; 21 °C	23 ± 2 °C	Average corresponds; variation not	Stricter variation in test due to lab climate conditions
Humidity	Low	P		50 ± 5% rH	No assessment possible
Inputs
Voltage	Low	E, P	207–253 V; 230 V	230 V ± 1%	Average corresponds; variation not	Rated voltage in Europe
Frequency	Low		49–51 Hz; 50 Hz	50 Hz ± 1%	Average and variation correspond	Rated frequency in Europe
Harmonic distortion	Low	E	Distorted	Plane sinus	No correspondence
User behaviour
Choice of nozzle - type - number	Medium High	E, P	Floor cleaning: consumers usually use one multipurpose (combination) nozzle. Less usage of special nozzles for special purposes. Above the floor cleaning: several nozzles used like upholstery nozzle, dusting brush, crevice tool	Partly covered in the standard: Floor cleaning: carpets, flat hard floors, hard floors with crevices Above the floor cleaning: fibre pick-up from upholstery	Floor cleaning: corresponds Above the floor cleaning: corresponds partly	Above the floor cleaning: far too many different kinds of potential usage
Cleaning head setting	Medium High	E P		Not included but under development (currently in EU guidance documents)	No assessment possible	Few data regarding customers’ behaviour regarding usage of various attachments and settings available
Motor settings	High	E, P	Full power	Maximum continuous airflow	Standard corresponded to practice in the past. Currently, more complexity due to extended instructions in the manual. More vacuum cleaners with automatic power settings on the market	Increasing complexity of vacuum cleaners makes it more and more difficult to find a proper definition in the standard to cover all potential settings and simultaneously not being in contradiction with the existing regulations (EC 2015)
Surface to be cleaned	High	P	Hard floor, carpet, upholstery	Flat hard floor, hard floor with crevices, carpet. 4 carpets specified, including the Wilton wool test carpet	Variation in carpets in principle captured but not implemented (single carpet used)	Use of single carpet due to cost and repeatability
Dust particle size	High	P		Between 0.09 and 0.2 mm	Low correspondence because no debris included	Discussion started to evaluate options with different types of dust like debris
Product features
Maximum usable volume of the dust receptacle	Medium	P		Not assessed

1. Average: EU average (unless mentioned otherwise)
2. ^aCategorical assessment: low, medium, high)
3. ^bRange: minimum-maximum (without extreme values)

Steps 1b and 2b: Listing of all performance aspects, selecting (most) relevant aspects, assessment of correspondence

Aspects of performance	Remarks	Assessment in practice	Assessment in standard
Dust pick-up	Measured with empty dust receptacle. Half-loaded receptacle test under preparation	Visual inspection.	Amount of dust in receptacle when test on carpet. On hard floor, dust amount left on floor is measured. Both after 5 double strokes
Motion resistance		Experienced force during operation	Force to move cleaning head measured
Fibre and thread removal		Visual inspection	Measured
Performance with loaded dust receptacle		Visual inspection	Measured
Filter water loss		Experienced.	Measured
Filtration efficiency			Amount of dust in the intake aerosol channel
Dust re-emission/filtration efficiency	A good evaluation of vacuum cleaner, not only filters used.	Highly relevant for allergy sufferers	Measured
Total emission while vacuum cleaning	Evaluation of total system (including attachments) used on a soiled floor	Highly relevant for allergy sufferers	Measured
Energy consumption
Energy consumption		Not assessed	Measured
Operational aspects
Ease of use / usability aspects		Experienced	Measured. Several tests: cleaning under furniture, radius of operation, flexibility of the hose, mass/dimensions, weight in hand
Noise		Experienced	Measured
Sustained performance		Experienced	Measured
Durability aspects		Experienced	Several durability tests on appliance and components (motor, hose, tube, cable reel, nozzles)

Step 3: Correspondence of assessment of performance

The main aspects of performance are assessed in the standard. However, the way in which these aspects are assessed does not include any feedback during the operation. Contrary in practice, a consumer gets visual feedback about dust pick-up, feels the force needed to move the head and will react on this. The dust re-emission is almost impossible to relate to consumer experience, because the particle sizes are chosen to reflect particles harmful for the health of the consumer. This is different for allergy sufferers—they experience directly dust emitted or raised by the vacuum cleaner.