Enhancing Productivity and Market Access for Key Staples in the EAC Region: An Economic Analysis of Biophysical and Market Potential

Abstract

In this chapter, we show how the current crop areas under three key staples—rice, maize, and beans—could be better aligned with the crop suitabilities that are inherent in the East African Community (EAC) region, through some key policy interventions. We take a multi-market model that was constructed for the 5 main countries in the EAC and use it to demonstrate how reducing transport costs, and increasing crop productivities can lead to market-level welfare improvements, as well as a closer alignment between the areas where the crops are cultivated, and the areas with the best agronomic suitability for those crops. At present, a significant proportion of those staples are grown in areas with limited growth potential, but opening up markets in combination with productivity-focused investments can allow countries to make better use of the crop potential they already have, and take advantage of regional market opportunities.

Technical Annex: Details of the EAC Multi-market Model

1.1 Annex 1: Structure of the Partial Equilibrium, Multi-market Model

The general schematic of a multi-market model is shown in Fig. 4 and encompasses a wide variety of models that are in common use in empirical economic work.

Fig. 4

Key relationships within a generic multi-market model

Figure 4 shows the basic conceptual outline of the class of economic model that we are building, following the theoretical treatment described in Sadoulet and De Janvry (1994). A model of agricultural markets has to take into account the markets for the products themselves—which we will mostly focus upon—as well as the markets for key factors that are necessary for production—chiefly labor and marketed inputs like fertilizer. In the case where agriculture is relatively low in input use and mostly uses un-paid household labor—which is common in the African context—then much of what is supplied to crop production may not be captured in this kind of framework and may not have observable data (without a detailed household-level production survey, which is beyond the scope of this study). In the case of high-value and highly commercialized crops, this may be possible—and agricultural may compete with other sectors for these inputs, in which case a multi-sector representation of input and output markets might be needed (as in many applied general equilibrium models). Our modeling work falls into the ‘partial equilibrium’ category—which signifies the fact that not all inputs and outputs within the economy are modeled, and the circular flow of revenue from production activities, to the consumer and the returns to the use of key inputs like capital are not accounted for fully. We have built upon the basic framework of a partial-equilibrium, multi-market model of agricultural trade described by Minot (2009). There are models of this class that have been built for many of the EAC countries—but they do not have the detail on agriculture that is needed for this project—so we have adopted this approach.

The key elements of Fig. 4 that we will take into account are:

• The assumption of profit-maximization on the part of agricultural producers.

• The assumption of utility maximization on the part of consumers of agricultural products.

• The influence of income growth on demand, although the growth in income will not be fully endogenized, as we do not capture the full picture of payments to households (from ag and non-ag activities) and their expenditure on food and non-food goods.

• We will consider the per-capita demand for products, and use exogenous projections of population growth to extrapolate it to the national-level demand.

The key components of the multi-market model are as follows:

• The supply, demand, and trade balance that occurs at the national level for each country in the EAC region

• The relationship that describes the response of crop production to price changes—in terms of irrigated and rainfed harvested area and yield

• The relationship that describes the commodity food demand to price and income changes

• The key relationships that link prices across regions and relate them to the transportation costs and the volume of trade between those regions.

These relationships comprise the essential components of the EAC multi-market model that we use in this study.

The overall balance between supply, demand, and trade is captured in this equation

$$\begin{aligned} & {\text{Supply}} + \left( {\sum\limits_{{{\text{other}}\;{\text{EACRegs}}}} {\left( {\text{inflows}} \right)} + {\text{iMport}}_{\text{RoW}} } \right) = {\text{Demand}} \\ & \quad+ \left( {{\text{eXprt}}_{\text{RoW}} + \sum\limits_{{{\text{other}}\;{\text{EACRegs}}}} {\left( {\text{outflows}} \right)} } \right) + {\text{StockChange}} \\ \end{aligned}$$

where ‘Supply’ is the aggregate production coming from the harvested area and yields across the sub-regions of each of the EAC countries, as is given here

$${\text{Supply}} = {\text{Area}} \cdot {\text{Yield}} = \sum\limits_{R} {\sum\limits_{s} {\left( {A_{R,s} \cdot Y_{R,s} } \right)} }$$

where each country is divided into sub-regions (provinces or districts) denoted by R which, in turn, are subdivided into zones of crop suitability that were defined by the GIS-based analysis of other WaLETS team members.¹² The multiplication of harvested area and yield and their addition over the sub-regions and crop suitability zones of each country is what constitutes the national production of each crop.

• Calibrating the supply side of the model

The yield of the crop, in each suitability zone, has been informed by the inputs of agronomic experts from each of the EAC regions covered in the WaLETS project. The information gained from these experts—which indicated the share of maximum potential yield of each crop that would be attainable within that zone—was used to allocate the base areas and yields of the model, for all the EAC countries, such that the national-level supply was matched, and could be balanced to demand and trade. This process is described in more detail, in the next subsection of this technical appendix.

The harvested area of each crop responds to prices, according to the net revenues that can be realized per hectare of each crop across the available cropland area. The key essence of this decision process is captured in the following optimization problem

$$\begin{array}{*{20}l} {\hbox{max} \quad \sum\limits_{j} {\left[ {p_{j,R} \cdot Y_{j,R,s} \cdot A_{j,R,s} - c\left( {A_{j,R,s} } \right)} \right]} } \hfill \\ {{\text{s}} . {\text{t}} .\quad \quad \sum\limits_{j} {A_{j,R,s} } \le \bar{A}_{s} } \hfill \\ \end{array}$$

where the regional prices of each crop (j) are $p_{j,R}^{{}}$, and $c\left( {A_{j,R,s} } \right)$ represents a nonlinear relationship between the total area harvested and the per-hectare cost of cultivation. This nonlinear relationship is a necessary component of calibrating the model to observed crop areas and is part of the ‘positive’ mathematical programming (PMP) approach (explained in the next subsection).

This method for allocating crop area is in contrast to the ‘reduced-form’ approach used in an earlier version of this work, in which an iso-elastic area response function was hypothesized ($A_{R,r}^{{}} = f_{A} (P) = c_{A}^{R,r} \cdot P^{\gamma }$¹³). In this function, the reaction of area to price changes reflects the behavior that is embodied in the optimization problem that we now model explicitly.

• The Positive Mathematical Programming Approach

The positive mathematical programming principle of Howitt (1995a, b)—referred to, popularly, as PMP—is a method of calibrating economic models of agricultural production, that exploits the mathematical principles of duality that is embedding in all mathematical programming models. In essence, the PMP approach imposes a degree of curvature upon the objective function of the mathematical programming model, such that it causes the model solution to exactly equate the implied marginal costs of land allocation that are reflected, implicitly, in the observed behavior of the decision-maker. Since we, typically, do not observe the marginal costs that the decision-maker faces—but, rather, the average costs that are reflected in the collected data—we often have trouble calibrating economic models to replicate the land allocation behavior that we observe from farmers.

The PMP approach takes the valuation of land (and other) resources that is implicit in the observed allocation of crop area and uses the ‘shadow values’ derived from a constrained ‘stage 1’ problem, in order to reconstruct the nonlinear cost function that allows the model to calibrate exactly to the observed data in ‘stage 2’.

To illustrate, let us suppose that we start with a mathematical programming problem of land allocation among alternative crops that is linear in both the revenue and cost terms—such as the following problem.

$$\begin{array}{*{20}l} {\hbox{max} \quad \sum\limits_{j} {\left[ {p_{j,R} \cdot Y_{j,R,s} \cdot A_{j,R,s} - c_{j,R,s} \cdot A_{j,R,s} } \right]} } \hfill \\ {{\text{s}} . {\text{t}} .\quad \quad \sum\limits_{j} {A_{j,R,s} } \le \bar{A}_{R,s} } \hfill \\ \end{array}$$

In this case, the parameter $c_{j,R,s}$ reflects the per-hectare cost of cultivation (specific to crop, region, and soil class) which remains constant at all scales of production activity. This is a classic linear programming type of problem which would tend to over-specialize in one activity—such that all of the available cropland would be allocated to the most profitable crop (i.e., the crop with the largest gross margin per hectare—$p_{j,R} \cdot Y_{j,R,s} - c_{j,R,s}$). This type of solution does not typically reflect the kind of behavior that one usually observes among farmers, where there is often a mix of crops in the farming portfolio. Since we only observe the average costs of production, per hectare ($c_{j,R,s}$), we seek to obtain a better measure of the actual marginal costs of production that an economically optimizing farmer would equate with marginal revenue when reaching the mixed allocation of cropland that we observe in data ($\left\{ {\bar{a}_{1} , \ldots ,\bar{a}_{k} } \right\}$ for k crop types).

Therefore, if we posit the existence of a nonlinear cost function in crop area that would allow the model to replicate these implicit (but un-observed) marginal costs of the quadratic form: $TC\left( {A_{j,R,s} } \right) = \phi_{j,R,s}^{0} \cdot A_{j,R,s} + \tfrac{1}{2}\phi_{j,R,s}^{1} \cdot \left( {A_{j,R,s} } \right)^{2}$.¹⁴

Then, we can use the PMP procedure to obtain the values of the parameters of this cost function in three separate stages.

In the first stage, we modify the linear programming problem, shown above, to include constraints on crop area, such that the model is forced to replicate the solution that we observe in the data

$$\begin{array}{*{20}l} {\hbox{max} \quad \sum\limits_{j} {\left[ {p_{j,R} \cdot Y_{j,R,s} \cdot A_{j,R,s} - c_{j,R,s} \cdot A_{j,R,s} } \right]} } \hfill \\ {{\text{s}} . {\text{t}} .\quad \quad \sum\limits_{j} {A_{j,R,s} } \le \bar{A}_{R,s} } \hfill \\ {\quad \quad \quad A_{j,R,s} \le A_{j,R,s}^{\text{obs}} \cdot \left( {1 + \varepsilon } \right)} \hfill \\ \end{array}$$

In the solution of the model, we expect (according to economic principle) for the marginal revenue of production to be equated to the marginal cost of production for each of the crops in the optimal allocation.

The ‘shadow value’ that comes from the calibration constraints of the constrained programming problem, above, at the optimal solution represents the difference between the marginal costs that the decision-maker is hypothesized to equate at the observed data point and the average costs which we observe in the data. In other words, ${\text{MC}}_{j,R,s} = {\text{AC}}_{j,R,s} + \lambda_{j,R,s}$, where MC and AC are the marginal and average costs per hectare of crop activity (j) in region R and on suitability class s. If we take our total cost function and manipulate it to obtain the functional form for the marginal and average cost, i.e.,

$$\begin{aligned} & \text{TC}\left( {A_{j,R,s} } \right) = \phi_{j,R,s}^{0} \cdot A_{j,R,s} + \tfrac{1}{2}\phi_{j,R,s}^{1} \cdot \left( {A_{j,R,s} } \right)^{2} \\ & \text{MC}\left( {A_{j,R,s} } \right) = \frac{{\partial \text{TC}\left( {A_{j,R,s} } \right)}}{{\partial A_{j,R,s} }} = \phi_{j,R,s}^{0} + \phi_{j,R,s}^{1} \cdot A_{j,R,s} \hfill \\ & \text{AC}\left( {A_{j,R,s} } \right) = \frac{{\text{TC}\left( {A_{j,R,s} } \right)}}{{A_{j,R,s} }} = \phi_{j,R,s}^{0} + \tfrac{1}{2}\phi_{j,R,s}^{1} \cdot A_{j,R,s} \\ & {\text{so that}} \\ & \text{MC} - \text{AC} = \lambda_{j,R,s} = \left( {\phi_{j,R,s}^{0} + \phi_{j,R,s}^{1} \cdot A_{j,R,s} } \right) - \left( {\phi_{j,R,s}^{0} + \tfrac{1}{2}\phi_{j,R,s}^{1} \cdot A_{j,R,s} } \right) \hfill \\ & \quad\quad\quad\quad = \tfrac{1}{2}\phi_{j,R,s}^{1} \cdot A_{j,R,s} \hfill \\ \end{aligned}$$

So we are able to recover the ‘slope’ of our cost function $\left( {\phi_{j,R,s}^{1} } \right)$ from the shadow values derived in ‘stage 1’, such that $\hat{\phi }_{j,R,s}^{1} = \frac{{2\lambda_{j,R,s} }}{{A_{j,R,s}^{obs} }}$. The ‘intercept’ of the quadratic cost function $\left( {\phi_{j,R,s}^{0} } \right)$ can then be obtained by using the average costs obtained from data $\left( {c_{j,R,s} } \right)$, and equating it to the functional form we derived earlier, such that:

$$\begin{aligned} & c_{j,R,s}^{\text{obs}} = {\text{AC}}\left( {A_{j,R,s} } \right) = \phi_{j,R,s}^{0} + \tfrac{1}{2}\hat{\phi }_{j,R,s}^{1} \cdot A_{j,R,s}^{\text{obs}} \\ & {\text{so}}\;{\text{that}} \\ & \hat{\phi }_{j,R,s}^{0} = c_{j,R,s}^{\text{obs}} - \tfrac{1}{2}\hat{\phi }_{j,R,s}^{1} \cdot A_{j,R,s}^{\text{obs}} \\ \end{aligned}$$

And we are able to obtain the values that define our cost function $\left( {\hat{\phi }_{j,R,s}^{0} ,\hat{\phi }_{j,R,s}^{1} } \right)$.

We can now insert this calibrated cost function into the mathematical programming problem, in place of the linear cost term, such that we obtain the following nonlinear programming problem

$$\begin{aligned} & \hbox{max} \sum\limits_{j} {\left[ {p_{j,R} \cdot Y_{j,R,s} \cdot A_{j,R,s} - \left( {\hat{\phi }_{j,R,s}^{0} \cdot A_{j,R,s} + \tfrac{1}{2}\hat{\phi }_{j,R,s}^{1} \cdot \left( {A_{j,R,s} } \right)^{2} } \right)} \right]} \hfill \\ & {\text{s.t.}} \sum\limits_{j} {A_{j,R,s} } \le \bar{A}_{R,s} \hfill \\ \end{aligned}$$

And solving this un-constraint, mathematical programming problem, with the nonlinear term for cultivated area embedded in it, will now allow the model to find an optimal solution that matches exactly to the observed crop areas observed in the data. This is the essence of the PMP methodology, as we have implemented it in our model.

The PMP calibration method has found wide-ranging applications to the construction of policy analysis models, and numerous examples of its use can be found in the work of various authors in the agricultural economics literature (Gohin and Chantreuil 1999; Barkaoui and Boutault 2000; Heckelei and Britz 2000, 2005; Heckelei and Wolff 2003; Heckelei et al. 2012; Judez et al. 2001; Röhm and Dabbert 2003; Henry de Frahan et al. 2007; Kanellopoulos et al. 2010; Howitt et al. 2012; Mérel and Bucaram 2010; Mérel et al. 2011, 2014; Mérel and Howitt 2014; Doole and Marsh 2014).

• Calibrating the demand side of the model with trade

On the demand side, we divide this between food and non-food demand—keeping non-food demand as an exogenous parameter $\left( {\bar{Q}_{D}^{\text{non-food}}} \right)$.

$${\text{Demand}} = f_{D} (P,y) \cdot {\text{popn}} + \bar{Q}_{D}^{\text{non-food}}$$

Like harvested area, food demand also responds to prices—albeit negatively, as we’d expect demand to go down with increasing prices. The per-capita demand function $f_{D} (P,y) = c_{\text{dmd}} \cdot P^{\varepsilon } \cdot y^{\eta }$ also has a response to per-capita income, which is generally positive for the three key crops we consider, but could be negative in other cases.¹⁵

The stock change at the national level is also kept fixed and exogenous, in our model. In principle, this can represent a component of supply response, as public and private managers of cereal stocks could decide to release stocks in response to price and augment national supply—or withdraw supply by adding to stocks. We do not have sufficient data on public and private stockholding, at the moment, to operationalize a behavioral model of this.

The price relationships lie at the heart of the way in which the model adjusts both demand and supply within each region—as well as the trade flows that happen across regions. The difference between prices in two adjacent regions determines the direction of the trade flow. The following relationship shows the ‘no-arbitrage’¹⁶ condition that is hypothesized to hold between two regions, where the transportation cost (TC) provides the ‘wedge’ between the two prices and accounts for the cost of delivering each unit of good from its origin to its destination

$$P_{{{\text{Region}}(R)}} + {\text{TC}}_{{R,R^{\prime}}} \ge P_{{{\text{Other}}\;{\text{Region}}(R^{\prime})}}$$

In this relationship, the price that one receives in the destination region $\left( {R^{\prime}} \right)$ should be no less than the purchase price in the region of origin (R) and the cost of transporting it between the two $\left( {{\text{TC}}_{{R,R^{\prime}}} } \right)$. Where the purchase + transport cost is greater, then it is not optimal to export from R to neighboring $R^{\prime}$(or, conversely, to import into $R^{\prime}$ from $R$). Where this relationship holds with equality (i.e., $P_{R} + {\text{TC}}_{{R,R^{\prime}}} = P_{{R^{\prime}}}$), then one could expect to have nonzero levels of trade. This complementary relationship¹⁷ between the levels of trade and the equality or non-equality of the no-arbitrage constraint, which can be written as

$$\begin{aligned} P_{R} + {\text{TC}}_{{R,R^{\prime}}} > P_{{R^{\prime}}} \quad {\text{and}}\quad {\text{Trade}}_{{R,R^{\prime}}} = 0 \hfill \\ P_{R} + {\text{TC}}_{{R,R^{\prime}}} = P_{{R^{\prime}}} \quad {\text{and}}\quad {\text{Trade}}_{{R,R^{\prime}}} \ge 0 \hfill \\ \end{aligned}$$

And it can, in turn, be summarized by the following equality statement

$$\left[ {P_{R} + {\text{TC}}_{{R,R^{\prime}}} = P_{{R^{\prime}}} } \right] \cdot {\text{Trade}}_{{R,R^{\prime}}} = 0$$

which must hold at all times. This type of relationship determines the trade levels between the EAC countries, as well as between each EAC country and the rest of the world.

In addition to this, there could be other taxes on imports or exports, such that the unit price of each traded good is affected—or there could be a quantitative limit on total exports and imports such that we have either a quota on imports

$${\text{Quota}}_{\text{Import}}^{R} \ge \sum\limits_{r} {M_{r}^{R} }$$

Or a quota on exports

$${\text{Quota}}_{\text{Export}}^{R} \ge \sum\limits_{r} {X_{r}^{R} }$$

where the quota (and tax) levels are decided by policy, and the levels of import $\left( {M_{r}^{R} } \right)$ and export $\left( {X_{r}^{R} } \right)$ for each country R are summed over the quantities going from each sub-region, r.¹⁸ In our model, we only apply import and export taxes (and quotas) on non-EAC trade, leaving the EAC region a ‘free-trade zone’, as it was intended.

In terms of defining the prices that exist on the ‘border’ and which are relevant to determining the levels of exports ($P_{X}$) to and imports ($P_{M}$) from the rest of the world, these can be adjusted from the fixed and exogenous ‘world price’ ($P_{\text{world}}$) to determine the import and export parity prices.

We can account for export taxes $\left( {t_{X} } \right)$ vis-à-vis FOB price that is relevant for exporters as

$$P_{X} = {\text{NER}} \times P_{\text{world}} \times \left( {1 - t_{X} } \right)$$

whereas the import taxes $\left( {t_{M} } \right)$ can be applied to the world price to define the relevant CIF price that an importer would care about

$$P_{M} = {\text{NER}} \times P_{\text{world}} \times \left( {1 + t_{M} } \right)$$

In each case, NER denotes the exchange rate between local currency and the US dollar (which the world price is denominated in).

In summary, the key endogenous variables that are solved by the model are:

$$X,M,P_{\text{region}}, {\text{inflows, outflows, Demand, Supply}}$$

whereas the exogenous and fixed parameters of the model are

$$P_{M} ,P_{X} ,P_{\text{world}} ,{\text{Quota}}_{\text{export}}^{R} ,{\text{Quota}}_{\text{import}}^{R} ,{\text{TC}},t_{M} ,t_{X} ,{\text{NER}}$$

The model is solved as a mixed complementarity problem—in which the following relationships must be satisfied by the optimal solution (i.e., the market equilibrium):

Export price relationship

$$\left[ {P_{R} + {\text{TC}} + \text{Im} {\text{Tax}}_{\text{export}} - P_{X} } \right] \cdot X = 0$$

Import price relationship

$$\left[ {P_{M} + {\text{TC}} + \text{Im} {\text{Tax}}_{\text{import}} - P_{R} } \right] \cdot M = 0$$

Domestic price relationships

$$\left[ {P_{R} + {\text{TC}}_{R,RR} - P_{{R^{\prime}}} } \right] \cdot {\text{TQ}}_{{R,R^{\prime}}} = 0$$

where ${\text{TQ}}_{{R,R^{\prime}}}$ represents the quantity traded between regions (going from R to $R^{\prime}$), such that

$\sum\nolimits_{{R^{\prime}}} {{\text{TQ}}_{{R^{\prime},R}} } = \sum\nolimits_{{{\text{other}}\;\text{Regs} }} {\left( {\text{inflows}} \right)}$ and $\sum\nolimits_{{R^{\prime}}} {{\text{TQ}}_{{R,R^{\prime}}} } = \sum\nolimits_{{{\text{other}}\;\text{Regs} }} {\left( {\text{outflows}} \right)}$ for any region R

Quota on exports

$$\left[ {{\text{Quota}}_{\text{Export}}^{R} - \sum\limits_{r} {X_{r}^{R} } } \right] \cdot \text{Im} {\text{Tax}}_{\text{export}} = 0$$

Quota on imports

$$\left[ {{\text{Quota}}_{\text{Import}}^{R} - \sum\limits_{r} {M_{r}^{R} } } \right] \cdot \text{Im} {\text{Tax}}_{\text{import}} = 0$$

where the implicit tax on exports or imports is nonzero if the quantity constraint on total exports or imports becomes binding. In such a case, the export and import price relationships have to account for nonzero values of both transportation costs and the implicit tax in the no-arbitrage condition.

This structure obviates the need for an explicit objective function (the maximization of joint consumer and producer surplus, for example)—but requires that the model have an equal number of equations and free variables. This requires us to exercise particular care in the preparation and checking of the data before use in the model.

• Calibration of model in Supply, Demand, and Trade

Based on the specified structure of the model, that we’ve described, and the data with which it has been built, we now have a complete framework for doing simulation of supply, demand, and trade in the EAC region. To ensure that the results are congruent with what we would expect, we must validate the base model results against observed data. As a first step in the model validation process, we verify that the model is able to reproduce the observed supply, demand, and trade that is observed in the data, in order to gain confidence in its proper functioning. This will verify that the calibration of transportation costs and border prices (with the rest of the World) were done correctly.

In Table 13, we show that the production and demand of the three key commodities in the EAC countries match the data from FAO exactly when the market equilibrium is simulated by the model.

Table 13 Calibration of model to observed production and consumption at the national level of each EAC country (‘000 tons)

Similarly, in Table 14, we show that the imports and exports of the three key commodities in the EAC countries also match the data found in the FAO trade balances.

Table 14 Calibration of model to observed trade at national level

Based on this, we can now have confidence that the national-level supply, demand, and trade balances can be captured correctly by the model, which allows us to proceed in carrying out alternative policy simulations. In addition to this level of calibration, we must also obtain sub-national values of area and yield for the three key staple commodities that adds up to the national-level totals for production that are shown (Table 13). This process of calibration is described in the next subsection.

1.2 Annex 2: Calibrating Base Areas and Yields to GIS-Based Modeling Outputs

In order to ensure that the model is completely consistent with the key underlying data, we had to undertake an extensive exercise to disaggregate the national-level production for each EAC country into sub-national areas and yields that are consistent with both (1) the administrative region-level statistics on irrigated and rainfed areas and yields provided by the HarvestChoice project database and (2) the division of total cropland into zones of suitability provided by the GIS analytical component of WaLETS.

These two sources of information provide the basis for calculating a plausible sub-national distribution of areas and yields for the three staple crops that meet these criteria:

• (1) They add up (across administrative units) to the national totals, reflected in the national-level supply—that must balance with total demand, stock change, and trade.

• (2) They add up (across suitability classes) to the same national totals.

• (3) They constitute a share of total cropland that is consistent with the GIS analytical outputs.

• (4) They reflect the relative yields across suitability classes that was suggested by the expert agronomists working on the WaLETS team.

In order to satisfy all of these criteria simultaneously, we had to use a model-based method to process the inputs from HarvestChoice, the WaLETS GIS team, and the agronomy experts, so that the outputs used for the economic model could reflect the best use of that information, and satisfy the necessary trade balance and adding-up conditions that are embedded in the trade model structure.

Since we are trying to obtain unique values for area and yield across sub-regions (r), crops (j), suitability classes (s), and countries (R), we end up solving a problem which is ill-posed. In other words, it contains more unknowns than data points and has negative degrees of freedom.

The maximum entropy formalism provides an efficient and robust way of solving ill-posed estimation problems and has been applied widely by scientists (ranging from the social to the physical sciences) to complex empirical problems.

• Maximum Entropy/Minimum Cross-Entropy Approach

In order to meet the challenge of simultaneously satisfying various balancing and adding-up constraints, while trying to get the estimates for shares of irrigated areas and yields in total area and production that are most consistent with the various (although not always compatible) sources of information that we have, requires an optimization-based approach. While a least-squares type curve-fitting approach can be used, in order to minimize the distance between a target level of ‘closeness’ to the existing data, while satisfying certain constraints that have to be met exactly can be used for problems like this, there is often the issue of how to satisfy several different targets simultaneously, without imposing undue weight on any particular objective criterion over another. The problem becomes particularly vexing when one is confronted with a relative scarcity of reliable data—which might be far fewer in number than the many unknowns which have to be determined. This situation leads to the type of ‘ill-posed’ problems, where the degrees of freedom are non-positive, and solving a classical inverse-type problem, where the unknowns may be solved for by linear algebraic inversion of a data matrix with respect to a vector of variables, becomes impossible.

In order to resolve this issue, we turn to cross-entropy-based techniques which are used to derive unknown distributions from fairly limited data and includes one’s own ‘prior’ beliefs on the underlying nature of the distribution, where possible. Cross-entropy methods have been used successfully in many types of statistical analyses in both the physical and social sciences and have even been used in IFPRI’s own work, such as balancing the social accounting matrix (SAM) of a computable general equilibrium model (Robinson et al. 2000), or in calculating the distribution of irrigated and rainfed crops based on global data from a variety of (sometimes inconsistent) datasets (You and Wood 2004).

The cross-entropy method is built upon the same ‘information-theoretic’ principles that underlie the principle of ‘maximum entropy’ that was introduced by Shannon (1948a, b) to describe the degree to which a distribution differs from a uniform and un-informative profile—thereby capturing the ‘surprise’ that is embodied in a (random) outcome. In juxtaposition to Shannon’s entropy measure $H = \sum\nolimits_{n}^{{}} { - p_{n} \log (p_{n} )}$ for $n$ discrete, random events, we can also express the cross-entropy of a distribution by the measure ${\text{CE}} = \sum\nolimits_{n} { - p_{n} \log \left( {\frac{{p_{n} }}{{\bar{p}_{n} }}} \right)}$, which includes the prior distribution of weights (or probabilities) $\left\{ {\bar{p}_{n} } \right\}$ that can be assigned for each random outcome. As shown by Kullback (1959), the maximization of the Shannon criterion with respect to the adding-up constraint $\sum\nolimits_{n} {p_{n} } = 1$ is equivalent to the minimization of the cross-entropy criterion, similarly constrained, if the prior distribution is uniform (i.e., assigns an equal likelihood to each outcome). The divergence of a calculated distribution from prior beliefs as calculated by the cross-entropy criterion conveys information content in a similar way to that calculated by the Shannon measure of information (Kullback and Leibler 1951).

• Implementation of Optimization Procedure

In applying this method to the problem of disaggregating areas and yields to the sub-national-level, based on the various sources of information that were available, we constructed an entropy-based estimation procedure which has the additional following features:

• It specifies the ‘shares’ of crop harvested area that belong to each suitability class, in each region—thus allowing them to act as ‘weights’ within the entropy framework.

• It specifies the shares of total cropland that belong to the suitability classes indicated in the outputs of the WaLETS GIS-analysis.

• It forces all shares to add up to one.

• It enforces all the adding-up relationships between total harvested crop and total cropland area that we would expect.

The key ‘known’ sources of data and ‘unknown’ variables that we have to solve for are summarized in Table 15.

Table 15 Key known and unknown variables in calibration model

Some of the ‘known’ parameters are outputs from the GIS modeling exercise and expert opinion (and, therefore, perhaps not 100% consistent with the underlying reality). But we treat these as inputs to the data processing program, which seeks to divide total cropland area into the area into:

• (1) the area occupied by the three key staples of interest and

• (2) the area occupied by all other crops that are not of interest to the modeling exercise.

Therefore, we seek to enforce the following identities:

$$\begin{aligned} A_{r,s}^{\text{Totcropland}} = \underbrace {{\sum\limits_{j} {A_{j,r,s} } }}_{\text{staples}} + A_{r,s}^{\text{Other}} \hfill \\ A_{r}^{\text{Totcropland}} = \underbrace {{\sum\limits_{j} {A_{j,r} } }}_{\text{staples}} + A_{r}^{\text{Other}} \hfill \\ A_{s}^{\text{Totcropland}} = \sum\limits_{r} {A_{r,s}^{\text{Totcropland}} } \hfill \\ \end{aligned}$$

where the quantities $A_{s}^{\text{Totcropland}} ,A_{r}^{\text{Totcropland}} ,A_{j,r}$ are known prior to the estimation, and the values of $A_{r,s}^{\text{Totcropland}} ,A_{r}^{\text{Other}} ,A_{r,s}^{\text{Other}} ,A_{j,r,s}$ must be solved for.

In order to disaggregate the areas of staples and croplands across the appropriate suitability classes, we must find the shares alpha beta, such that the following relationships hold

$$\begin{aligned} &A_{j,r,s} = \alpha_{j,r,s} \cdot A_{j,r} \quad \quad \quad \quad \quad \alpha_{j,r,s} \in \left( {0,1} \right] \hfill \\ &A_{r,s}^{\text{Totcropland}} = \beta_{r,s} \cdot A_{r}^{\text{Totcropland}} \quad \quad \beta_{r,s} \in \left( {0,1} \right] \hfill \\ &\sum\limits_{{s \in S_{j} }} {\alpha_{j,r,s} } = 1 \hfill \\ \end{aligned}$$

where we carry out the summation of shares for staple crops ($\sum\nolimits_{{s \in S_{j} }} {\alpha_{j,r,s} } = 1$) only over a defined sub-set of suitabilities that match with the crop type. In other words, we don’t sum values for maize over bean suitabilties, and likewise for others. Since we know that the land areas defined by the maize, bean, and rice suitabilties overlap strongly with each other, we do not enforce the summation over the ‘beta’ values (i.e., $\,\sum\nolimits_{s} {\beta_{r,s} } = 1$).

Since we must also enforce the adding up of total production to the national and sub-national totals provided by the multiplication of areas and yields, across sub-regions and suitability classes, we must consider these identities

$$\begin{aligned} &\sum\limits_{r} {Q_{j,r} } = \sum\limits_{s} {\sum\limits_{r} {A_{j,r,s} \cdot Y_{j,r,s} } } \hfill \\ &Y_{j,r,s} = Y_{j}^{\text{MaxPotential}} \cdot \theta_{s}^{\text{attain}} \cdot \Delta_{j,r,s} \quad \theta_{s}^{\text{attain}} \in \left( {0,1} \right],\Delta_{j,r,s} \in \left( {0, + \infty } \right) \hfill \\ \end{aligned}$$

where we allow the sub-regional, suitability-specific yields to be informed by expert opinion, but ultimately allowed to deviate from it to the extent necessary to make the overall balance of area and production hold. The country-level agronomy experts from the WaLETS team described the maximum potential yield for each crop in each country $\left( {Y_{j}^{\text{MaxPotential}} } \right)$, and the share of that maximum potential that is attainable in each suitability class $\left( {\theta_{s}^{\text{attain}} } \right)$. We allow the yields we ultimately solve for to deviate from that amount by an allowable margin $\left( {\Delta_{j,r,s} } \right)$ that we try and keep as close to 1 as possible—so as to preserve as much of the expert information as possible. So, we place a ‘penalty’ upon any deviations in the value of $\Delta_{j,r,s}$ from one, using the cross-entropy principle in the objective function of the estimation program.

Now that we’ve described the key components of the program, we can now write out the entropy-based mathematical programming problem we are trying to solve as follows:

$$\begin{aligned} &\mathop {\max}\limits_{\begin{subarray}{l} \alpha_{j,r,s} ,\beta_{r,s} ,Y_{j,r,s} ,\Delta_{j} \\ A_{j,r,s} ,A_{r,s}^{\text{Totcropland}} \\ A_{r,s}^{\text{Other}} ,A_{r}^{\text{Other}} \end{subarray} } \sum\limits_{j} {\sum\limits_{r} {\sum\limits_{s} {\left[ { - \alpha_{j,r,s} \cdot \log \left( {\alpha_{j,r,s} } \right)} \right]} } } + \sum\limits_{j} {\sum\limits_{r} {\sum\limits_{s} {\left[ { - \beta_{r,s} \cdot \log \left( {\beta_{r,s} } \right)} \right]} } } \\ & \quad + \sum\limits_{j} {\left[ {\Delta_{j} \cdot \log \left( {{\raise0.7ex\hbox{${\Delta_{j} }$} \!\mathord{\left/ {\vphantom {{\Delta_{j} } 1}}\right.\kern-0pt} \!\lower0.7ex\hbox{$1$}}} \right)} \right]} \hfill \\ & {\text{s.t.}} \hfill \\ &\sum\limits_{r} {Q_{j,r} } = \sum\limits_{s} {\sum\limits_{r} {A_{j,r,s} \cdot Y_{j,r,s} } } \hfill \\ &Y_{j,r,s} = Y_{j}^{\text{MaxPotential}} \cdot \theta_{s}^{\text{attain}} \cdot \Delta_{j} \hfill \\ &A_{r,s}^{\text{Totcropland}} = \underbrace {{\sum\limits_{j} {A_{j,r,s} } }}_{\text{staples}} + A_{r,s}^{\text{Other}} \hfill \\ &A_{r}^{\text{Totcropland}} = \underbrace {{\sum\limits_{j} {A_{j,r} } }}_{\text{staples}} + A_{r}^{\text{Other}} \hfill \\ &A_{s}^{\text{Totcropland}} = \sum\limits_{r} {A_{r,s}^{\text{Totcropland}} } \hfill \\ & \hfill \\ &A_{{j,r,s}} = \alpha _{{j,r,s}} \cdot A_{{j,r}} \\ & A_{{r,s}}^{{{\text{Totcropland}}}} = \beta _{{r,s}} \cdot A_{r}^{{{\text{Totcropland}}}} \\ & \sum\limits_{{s \in S_{j} }} {\alpha _{{j,r,s}} } = 1, \\ & \alpha _{{j,r,s}} \in \left( {0,1} \right],\beta _{{r,s}} \in \left( {0,1} \right],\theta _{s}^{{{\text{attain}}}} \in \left( {0,1} \right],\Delta _{j} \in \left( {0, + \infty } \right) \end{aligned}$$

where the objective function is a hybrid of the maximum entropy problem (with respect to the $\alpha_{j,r,s} ,\beta_{r,s}$ variables and a cross-entropy problem (with respect to the $\Delta_{j}$ variable).

This program is run for the existing data in order to obtain a distribution of areas and yields over each country’s sub-regions and suitability classes that can be used as base data for the economic market model. Given that production was constrained, within the program, to match the national-level production of the three staples, the model will replicate the base-year supply, demand, and trade when it is simulated.

As better information becomes available—either from the GIS-based analysis or the yield potential assessments of the agronomy experts—this data can be put into the calibration model so that it can be rerun and generate a new base data set of disaggregated areas and yields.

Table 16 shows the results from the model—aggregated across administrative sub-regions for each country—to display in a more convenient fashion.

Table 16 Calibration of base area and yield levels across suitability classes for each EAC country