UICC Tobacco Control Fact Sheet No. 19

Evaluation of Anti-Smoking Programmes

Introduction

Few anti-smoking programmes have undergone specific evaluation in developed countries; the number is even lower in developing countries.

Figures sometimes suggested for evaluation may be as high as 25% of a programme's budget. Yet not all evaluation need be elaborate, expensive or overly time-consuming.

No country can justify delaying tobacco control measures while lengthy and costly research is carried out. Sufficient worldwide scientific data already exist for all countries to take immediate action against tobacco.

However, it is preferable for each country first, or early in the epidemic, to collect baseline data on smoking prevalence (or estimate consumption from sales of tobacco products), determine smoking-related mortality and morbidity, and analyze the costs of smoking to the economy.

Research can help determine the scope of the problem, identify the most effective anti-smoking measures, and illustrate to governments and other potential aid-givers the need for continued support and funding.

The first measure of success in tobacco control is the degree to which key groups have been persuaded that smoking is harmful and that public health action should be taken. This can be measured as follows:

Health profession
by their stand, statements, financial commitment, and actions;

Government
by legislative and administrative action, voting record, speeches, and funding to tobacco control;

Media
by column inches given to responsible reporting and editorial comment on the tobacco epidemic.

Evaluation

There are four traditional stages of evaluation. Not all countries will be able to measure all of the components, but some form of evaluation can be included in almost any size of budget.

University medical, teaching, statistics or legal departments can be encouraged to assist in evaluation of programmes through postgraduate or undergraduate projects.

1. Formative evaluation

Includes assessment of the goals, identification of the target groups, pre-testing, and cost evaluation.

• Goals and target groups
  Goals may include changes in knowledge, attitudes, or practices. Target groups may be the young, the still-healthy adult smoker, the damaged smoker or even the adult non-smoker. Goals should be realistic and specific, not 'reduction in smoking.' Sometimes interim goals should be identified. Goals may be the continuation of the status quo, such as maintaining low female smoking rates in developing countries.
• Pre-testing is the systematic gathering of target audiences' reactions to messages and materials before they are produced in final form. Pre-testing can assess whether the message commands attention, is comprehended, is relevant to the target audience, is believed, and is acceptable.
• Evaluation of cost-effectiveness
  Tobacco prevention involves more efficient use of funds per unit of health gained than many other programmes. Many activities cost nothing, such as giving newsworthy stories to the media or talks to schools, or by debating with the tobacco industry on radio and television.

2. Process evaluation

examines procedures and tasks involved in implementing a programme, such as assessing the quantity of materials printed and distributed or the number of people reached.

3. Outcome evaluation

includes short- and medium-term results, such as changes in knowledge, attitudes, intentions (for example, to quit smoking), or in smoking prevalence or tobacco sales. Preferably these changes are compared to a control group.

Because many countries take several measures simultaneously, it is sometimes difficult to quantify the specific effects of each. It is particularly difficult to isolate a control group in mass media strategies.

• Health information and education campaigns
  Of all the initiatives on tobacco prevention, mass media actions, including no-smoking days and quit campaigns, are evaluated most often.
• Price policy

Charting price increases with consumption trends shows that tobacco use declines as its price increases.

• Ban on promotion
  Detailed surveys of prevalence among youth show the effectiveness of promotional bans. In 1987, the Australian State of Victoria placed a 5% levy on tobacco to establish an independent Health Promotion Foundation, VicHealth, offering funding for health promotion, and alternative funding to sports and arts. The success of this initiative could be assessed when, after the first year of operation, only three sports organizations continued to accept tobacco sponsorship. Success can also be gauged by several other governments copying the VicHealth model.
• Health warnings
  Evaluation is by awareness of the warnings or their readability on outdoor advertising. The tobacco industry provides indirect evaluation: the only words in health warnings the industry seriously opposes are 'addiction' and 'kill,' a useful indication for countries to include these words.
• Ban on sales to youth
  Ban on sales to minors may be evaluated using children and adolescents in compliance checks, often referred to as "sting operations". Evaluation in some industrialized countries shows that merchant education accompanied by regular compliance checks reduces sales to minors as well as smoking initiation among school children. However, enforcement is minimal or non-existent in many developing countries, especially where single cigarettes are sold by street vendors.
• Ban on new forms of tobacco
  Evaluation of bans on manufacture, importation and sale of smokeless tobacco is by compliance with the law and non-use of the product.
• Smoke-free areas
  Success may be evaluated by implementation, compliance, harmony of implementation, attitudes and norms, and smoking behaviour.
• Litigation
  Litigation may not only be evaluated by the winning of a case, but also by the publicity it generates, or even by the effect on tobacco shares on the stock markets.
• Medical intervention trials
  Large-scale medical intervention trials show that reducing smoking leads to a decrease in lung cancer, heart disease, and other diseases. Designing, conducting, and evaluating multiple risk-factor intervention trials, such as those for preventing coronary heart disease, are more complex.

4. Impact evaluation

by prospective studies focuses on the long-term effect of the programme by measuring changes in disease and death.

Indirect evaluation by the tobacco industry

Health advocates can make use of sophisticated evaluation funded by the tobacco industry. As a general rule, the more strongly the tobacco industry opposes an anti-smoking measure, the more useful the measure is, and the more health advocates should concentrate their efforts in these areas.

The tobacco industry concentrates its opposition on two areas of intervention: bans on tobacco promotion (and the establishment of a VicHealth model) and tax increases.

The tobacco industry supports weak and unenforceable bans on sales to children. It has never opposed health education in schools, suggesting the lack of effectiveness of these two measures.

The establishment of a Tobacco Institute by a consortium of tobacco manufacturers is a good indication that the health forces in that country are being effective.

Where to find data for use in evaluation

Existing information may be obtained, generally free of charge, from the following sources:

Ministry of Health Prevalence and health data; national policy and action

Census and Statistics Department Population demography

Ministry of Law Tobacco legislation

Ministry of Education Health education in schools

Ministry of Finance Tobacco tax; total tax

Ministry of Trade and Industry Import/export of tobacco

Ministry of Agriculture Tobacco acreage; numbers of tobacco farmers

Police Department Cigarette smuggling; black market information;
counterfeit cigarettes

Fire Department Fires caused by smoking

Births and Deaths Registry Population statistics

Transport companies Their policy on smoking

Private companies Workplace tobacco policy

Universities Research data on tobacco-related death and disease;
public opinion surveys

Medical societies Disease data

Cancer and heart disease registries Tobacco-related mortality

Sports/arts bodies Tobacco sponsorship

Newspapers Articles on tobacco; interviews with tobacco executives

Regional organizations Action taken by other countries

Media and survey research firms Tobacco advertising data

Advertising industry Tobacco advertisements

Journals of tobacco industry Trade and other data

Journals of advertising industry Tobacco advertising data

International health organizations Global information on tobacco

e.g. UICC and WHO

Medical journals Articles on tobacco, (NB "Tobacco Control")

Tobacco industry Published data in their trade journals; annual reports

Food and Agriculture Organization Tobacco growing acreage

Acknowledgement

This fact sheet has been prepared for the UICC by Dr. Judith Mackay, Asian Consultancy on Tobacco Control, Hong Kong, and Dr. Ronald M. Davis, Director, Center for Health Promotion and Disease Prevention, Henry Ford Health System, Detroit, MI, USA.

Shortened from: Mackay, J. and Davis, R.M., "Assessing Community Interventions to reduce smoking". International Journal of Technology Assessment in Health Care, 7:3 (1991), 345-353.

4/1996

Tobacco and Cancer Programme International Union Against Cancer 3, Rue du Conseil-Général, 1205 Geneva, Switzerland Tel: (4122) 809 1830, Fax: (4122) 809 1810 E-mail: tobacco-control@globalink.org | Tobacco and Cancer Programme