Extracting government subsidy data, not an easy endeavour

Data collected with has been explored with Silk, a tool that displays interactive charts, maps and web pages. How we got the data is a story in itself.

On May 21, 2014, responding to questions rose in parliament, the Ministry of Finance published overviews of subsidies awarded in 2013 by each government department. It involved more than 21 thousand grants. Since government departments often do not know what grants other departments award, we have put the data in a structured form together. On June 12, 2014, Open State Foundation launched de web tool to make these subsidies and recipients searchable.

We noticed that the Ministry of Economic Affairs provided most of the subsidies. In 2013 15671 recipients received grants from the various government departments. Housing corporations received the most subsidies and research institutions received the largest subsidies.

In the Netherlands there are also 12 regional governments. Open State Foundation took the challenge to see whether we would be able to disclose all subsidies granted in 2013. At the moment we collected the data of all government departments and all provinces. The collection of information was not so simple as it seemed to be.

Most provinces with a grant registry published the information in a PDF file. Only two regional governments also made the original Excel file available. And another two provinces did not had a grant registry at all (eventually we filed a freedom of information request to obtain the overviews). When asked whether this information could be published in an open format, such as csv or the source file, provinces responded quite differently. Also not all provinces provided the same information. Some published all the grants they rewarded; others published only those above a certain threshold.

Also gathering European subsidies granted in the Netherlands was a difficult endeavor. The Financial Transparency System only includes subsidies granted by the European Commission directly. That is only 20 percent of all European subsidies awarded in the Netherlands. Also a website that claims transparency on European subsidies granted in the Netherlands – – did not include all of them. Participation in that portal was on a voluntary basis.

So we had to go to the national and regional authorities that have been tasked with delivery of these subsidies. We filed a freedom of information request with the Ministry of Economic Affairs on July 3, 2014 and requested a list of all grants made in 2013 from European Regional Development Fund, European Social Fund and Cohesion Fund. Additionally, we scraped data on the grants awarded from the European Agricultural Fund for Rural Development and the European Maritime and Fisheries Fund which are in a search engine of the Netherlands Enterprise Agency.

For information on subsidies and recipients from municipalities it was even more difficult. We have at the moment data from only 27 municipalities with an online subsidy register. All records were PDF files. We extracted the data out of the PDF files to have them included in Even though, information on public grants awarded might seem very basic, unfortunately collecting this data in a sustainable way still takes time. This is the moment when change needs to happen.