%load_ext autoreload
%autoreload 2
This site provides an extensive list of match statistics, outcomes and odds. However, for some reason there is no easy way to download the data. The main page contains links to the main leagues. Each main league site then provides a list of links to each result csv, grouped by season and league.
mainpage_link = r'https://www.football-data.co.uk/data.php'
mainpage_bs = cache(mainpage_link, 'football_data_mainpage')
Since parallelizatoin doesn't really work from within jupyter notebooks (at least on windows), I'll use this notebook to develop all necessary functions to later run a normal python script (with parallelization).
The column names for this datasource are rather cryptic. Here are the most useful ones:
- Date
- HomeTeam, AwayTeam
- FTHG*, FTAG** -> Full Time Home/Away Team Goals
- B365H/D/A Bet365 home/draw/away odds
The following two sections derive the method for generating all necessary download links to the individual csv files. Most of this derivation is not needed in the actual download script and is only kept here for future reference.
The sub-pages we are looking for have sub-urls of roughly the shape: \<country_name>m.php, so lets look for links like this.
php_links = find_links_by_func(mainpage_bs)
print(f'Number of php links: {len(php_links)}')
print('Examples:')
print('\n'.join(php_links[65:75]))
So the good news is that the links we're looking for are in there. The bad news is that there's also a bunch of other stuff. Lets try using some regex magic.
country_links = find_links_by_pattern(mainpage_bs, r'\w+m\.php')
country_links
Manually checking the site shows that this is exactly the list of links we're looking for. So far so good.
The country links we gathered in the last step are relative. To build the full links we have to join them with the base link.
example_country_link = urllib.parse.urljoin(mainpage_link, country_links[0])
country_html = cache(example_country_link, 'football_data_country_html')
example_country_link
The result files seem to be located all under the same structure: https://www.football-data.co.uk/mmz4281/
The next part that follows is the season, encoded as yyYY. Here yy are the last two digits of the year of the seasons beginning. YY are the last two digits of the year of the seasons ending.
We'll try to parse all available years from the example country page:
csv_links = find_links_by_pattern(country_html, 'mmz4281/\d{4}/\w*\d\.csv')
csv_links[:5]
From here we can easily extract the season data:
seasons = set(csv_link.split('/')[1] for csv_link in csv_links)
print(f'Amount of seasons: {len(seasons)}')
print(f'Example: {seasons.__iter__().__next__()}')
Lets refactor that as preparation for using for all the countries:
def unique_seasons(html):
'Returns a list of all unique seasons for which result.csv files are found in the given html'
csv_links = find_links_by_pattern(html, 'mmz4281/\d{4}/\w*\d\.csv')
return set(csv_link.split('/')[1] for csv_link in csv_links)
assert unique_seasons(country_html)==seasons
country_links = [urllib.parse.urljoin(mainpage_link, country_link) for country_link in country_links]
seasons = []
for country_link in country_links[1:]:
print('Processing ', country_link)
html = get_html(country_link)
unique = unique_seasons(html)
if unique:
seasons.append(unique)
else:
print('Found no seasons')
Let's see which seasons are available for all leagues:
seasons = list(functools.reduce(lambda a,b: a.intersection(b), seasons))
print('Number of available complete leagues:', len(seasons))
print(', '.join(seasons))
So appearently the data goes from 1995/96 all the way up to 2019/2020 (the currently running season). I'll grab all the data that's available, we can sort it out later.
Looking at the csv links from earlier, the next part is a sub-league code, in this case 'E0'.
csv_links[0]
The letter(s) in this code refer to the country in which the season is played. The number seems to be a sort-of ranking. For most countries, number 1 is assigned to the highest league. But not for all. In the example above for england, number 0 is assigned to the premier league. We'll therefore try to get all the relevant subleague-number -> subleague-name mappings.
We still have the example html code some country page.
Since we know that each country page has links to the last season (18/19), we'll parse for result links for this season:
def find_subpage_mapping(country_html):
for link in find_links_by_pattern(country_html, r'mmz4281/1819/\w*\d\.csv', return_href=False):
match = re.match(r'mmz4281/1819/(\w*\d)\.csv', link.get('href'))
print(f'{match.group(1)}: {link.text}')
find_subpage_mapping(country_html)
for country_link in country_links:
country = re.search('co.uk/(.+)\.php', country_link).group(1)
html = cache(country_link, f'football_data_{country}')
print('-'*20)
print(country_link)
find_subpage_mapping(html)
print('-'*20)
We're interested in the highest league for each country, we'll manually grab those codes:
We have all the seasons and league information:
FOOTBALL_DATA_LEAGUES
FOOTBALL_DATA_SEASONS
print('Number of leagues: ', len(FOOTBALL_DATA_LEAGUES))
print('Number of seasons for each league: ', len(FOOTBALL_DATA_SEASONS))
For a given league and season
season = FOOTBALL_DATA_SEASONS[5]
league_code = FOOTBALL_DATA_LEAGUES[7].code
print('Season: ', season)
print('League code: ', league_code)
we can generate the link to the results file:
build_full_link(season, league_code, mainpage_link)
A team of researchers derived a profitable method for sports betting by taking averages of odds from different bookkeepers. They hosted their dataset on kaggle:
https://www.kaggle.com/austro/beat-the-bookie-worldwide-football-dataset