98 million fields and 12 crops. How we made the OneSoil Map
The interactive map contains information on fields and crops in 59 countries. For five years. Read the story behind its creation.
Reading time: 7 minutes
In October 2018, we launched the interactive OneSoil map that provides information about every field in the USA and Europe. Since then numerous media have published articles about the map and we have received hundreds of emails with questions and offers. On Product Hunt, OneSoil map has collected almost 1,500 upvotes; this is unprecedented for an AgTech product.
In August 2022, we released an updated OneSoil Map with expanded geography and improved data accuracy.
In August 2022, we released an updated OneSoil Map with expanded geography and improved data accuracy.
What is the OneSoil Map
Like all OneSoil products, the interactive map is powered by machine learning algorithms and satellite imagery. It contains information on 98 million fields and 12 major crops in 59 countries. Companies that need agricultural analytics can buy any data from 2018 through 2022. Sandboxes of the 2020 map are available for select regions on all continents.
With the map, you can track trends at the country and regional level. For example, you can find out how much acreage was corn in the US in 2020 (35.5 million hectares) or which region of Belgium has the most wheat fields (Wallonia).
With the map, you can track trends at the country and regional level. For example, you can find out how much acreage was corn in the US in 2020 (35.5 million hectares) or which region of Belgium has the most wheat fields (Wallonia).
With the map, you can also find out information about an individual field: its size, culture, chart of plant development, and a field score. This measure is calculated by the NDVI index, climatic indicators, and relative field productivity. At any time, you can see how your beet field is doing or check what area your neighbor has planted with legumes last year. In addition to all this, the map is simply beautiful. We added a button called "random beautiful fields": it transports you between 35+ places around the world, each resembling a piece of abstract art.
Data: to collect, to process, to compress
For the OneSoil Map, we used photos shot by the Sentinel-2 satellite of the European Union’s Copernicus programme. A total of 250 Tb of data was processed for Europe and the USA. At the first stage, we preprocessed images: we cleaned the clouds, shadows, snow, and compressed the data down to 50 Tb. Then we started to search for field boundaries and classify cultures with our machine learning algorithms. At the output, we received about 250 Gb of vector maps with field geometries and cultures.
All this data was then transferred to our back-end developer Evgeny Voronets. "For processing data and calculating statistics, I used the PostgreSQL database with the PostGIS extension. After exporting the original vector data, I received a database with about 180 million records on field geometry and another billion of records of additional attribute information for three years", says Zhenya. Using this amount of data, we calculated statistics, ratings, and determined the popularity of cultures in different regions of the world. All this information is displayed in the left column and hovers on the OneSoil Map. v
All this data was then transferred to our back-end developer Evgeny Voronets. "For processing data and calculating statistics, I used the PostgreSQL database with the PostGIS extension. After exporting the original vector data, I received a database with about 180 million records on field geometry and another billion of records of additional attribute information for three years", says Zhenya. Using this amount of data, we calculated statistics, ratings, and determined the popularity of cultures in different regions of the world. All this information is displayed in the left column and hovers on the OneSoil Map. v
We wanted to calculate and display agronomic indicators across the field quickly, as well as visualize plant development over the season. For this, we used our own approach to caching and compressing satellite data. This made it possible to reduce the size of the data storage by 100−200 times and obtain field information within 1 second. We also added a field score to this version of the OneSoil Map. This measure allows you to quickly evaluate a field rating. "Field score is the first step towards yield forecasting, which our team is currently working on," - explains Data Scientist Alexander Kalinovsky.
Map: to select the format and to prepare the data
In order to visualize the data, we eventually used the Mapbox service. In general, there are two approaches to creating a map; we tried both. The first is to create a raster map. "In this case, we divide the map into squares, which we then render into pictures and store on the server. The browser loads several images and moves them when the user explores the map," says front-end developer Dmitriy Kabak. This approach allows you to display all the fields, without the need to filter anything. The end result is beautiful but such a map is static. Plus, raster images weigh quite a lot.
The second approach is to create a vector map. "The browser loads vector data and animates it on the client side. This is how modern Google and Yandex maps work. The data weighs less than the pictures, and allows you to change the design of any element," explains Dima. The Mapbox service allows creating such a map. In particular, their Mapbox GL library is an open source tool for displaying maps on the web. Amongst other things, Mapbox provides paid data storage service. You can manually upload your data to their servers, and Mapbox will quickly distribute information, ensuring accurate map operation. This is a significant part of the work, and due to the fact that Mapbox takes on this function, our task was significantly simplified.
The second approach is to create a vector map. "The browser loads vector data and animates it on the client side. This is how modern Google and Yandex maps work. The data weighs less than the pictures, and allows you to change the design of any element," explains Dima. The Mapbox service allows creating such a map. In particular, their Mapbox GL library is an open source tool for displaying maps on the web. Amongst other things, Mapbox provides paid data storage service. You can manually upload your data to their servers, and Mapbox will quickly distribute information, ensuring accurate map operation. This is a significant part of the work, and due to the fact that Mapbox takes on this function, our task was significantly simplified.
Wrap up
During development, the OneSoil Map concept changed a lot. At first, we planned to make a simple visualization of fields and cultures around the world. The final product turned out to be much more complex: there are dozens of statistics, ratings, and view modes. Hundreds of investors, foundations, and academic researchers have written to us since the 2018 release. Also, we will use the found technological solutions, such as the Next JS framework and the Tippecanoe utility, in further work on the OneSoil platform.
We are the first ones to map all the fields in the USA and Europe covering the last three years.
We are the first ones to map all the fields in the USA and Europe covering the last three years.
Do you like this post?
Related Articles
Comments