Should you catch your umbrella before going out? The verification of weather forecasts will only be useful if these forecasts are accurate.
Spatial prediction problems, such as weather forecasts or estimation of air pollution, imply predicting the value of a variable in a new location based on values known in other places. Scientists generally use proven validation methods to determine the quantity to trust these predictions.
But MIT researchers have shown that these popular validation methods can fail very badly for space prediction tasks. This could lead someone to believe that a forecast is correct or that a new prediction method is effective, while in reality it is not the case.
Researchers have developed a technique to assess prediction validation methods and used it to prove that two conventional methods can be substantially wrong on spatial problems. They then determined why these methods can fail and created a new method designed to manage the types of data used for space forecasts.
In experiences with real and simulated data, their new method has provided more precise validations than the two most common techniques. The researchers evaluated each method using realistic spatial problems, in particular by predicting wind speed at Chicago O-Hare airport and providing for air temperature with five American metro locations.
Their validation method could be applied to a range of problems, from the help of climatologists to predict sea surface temperatures to help epidemiologists to estimate the effects of air pollution on certain diseases.
“Hopefully this will lead to more reliable assessments when people offer new predictive methods and better understanding of methods,” explains Tamara Broderick, associate professor of the Department of Electrical and Computer Science (EECS), member of the Information Laboratory and Decision Systems and Istituated for Data, Systems and Society, and an Affiliate of IT and CS).
Broderick is joined on the paper by the main author and the Postdoc David R. Burt and the student graduated from the EECS Yunyi Shen. Research will be presented at the International Conference on Artificial Intelligence and Statistics.
Validation assessment
The Broderick group has recently collaborated with oceanographers and atmospheric scientists to develop automatic learning prediction models that can be used for problems with a strong space component.
Thanks to this work, they have noticed that traditional validation methods can be inaccurate in space. These methods maintain a small amount of training data, called validation data and use them to assess the precision of the predictor.
To find the root of the problem, they carried out an in -depth analysis and determined that traditional methods make hypotheses inappropriate for spatial data. The evaluation methods are based on hypotheses on how the validation data and the data that we wish to provide, called test data, are linked.
Traditional methods assume that validation data and test data are independent and distributed identically, which implies that the value of any data point does not depend on the other data points. But in a spatial application, this is often not the case.
For example, a scientist can use validation data of EPA air pollution sensors to test the accuracy of a method that predicts air pollution in conservation areas. However, EPA sensors are not independent – they were located according to the site of other sensors.
In addition, perhaps validation data comes from EPA sensors near cities while conservation sites are in rural areas. Because this data comes from different places, they probably have different statistical properties, so they are not distributed identically.
“Our experiences have shown that you get really bad answers in the spatial case when these hypotheses formulated by the validation method are decomposed,” explains Broderick.
The researchers had to offer a new hypothesis.
Specifically spatial
By thinking specifically of a spatial context, where the data is collected from different places, they have designed a method which assumes that validation data and test data vary gently in space.
For example, it is unlikely that air pollution levels are considerably changing between two neighboring houses.
“This regularity hypothesis is appropriate for many spatial processes, and it allows us to create a means of assessing space predictors in the spatial field. To our knowledge, no one has made a systematic theoretical evaluation of what was wrong to find a better approach, ”explains Broderick.
To use their evaluation technique, we would enter its predictor, the locations they wish to predict and their validation data, then it automatically does the rest. In the end, he considers how precise the predictor's forecasts will be for the location in question. However, the effective evaluation of their validation technique has proven to be a challenge.
“We do not assess a method, we rather assess an assessment. So we had to step back, think carefully and show creativity on the appropriate experiences that we could use, ”explains Broderick.
First of all, they designed several tests using simulated data, which had unrealistic aspects but allowed them to carefully control the key parameters. Then they created more realistic semi-emotional data by modifying the real data. Finally, they used real data for several experiences.
The use of three types of data from realistic problems, as predict the price of an apartment in England according to its location and the forecast of wind speed, allowed them to carry out a complete evaluation. In most experiences, their technique was more precise than the traditional method to which they compared it.
In the future, researchers plan to apply these techniques to improve the quantification of uncertainty in space. They also want to find other areas where the regularity hypothesis could improve the performance of predictors, such as with data from the chronological series.
This research is funded, in part, by the National Science Foundation and the Office of Naval Research.
