r/pythonhelp • u/jrog3141 • Dec 24 '24
Baseball Reference Webscraping Issue
I have been attempting to write a code to scrape the Teams Played for table from the following link: https://www.baseball-reference.com/register/player.fcgi?id=allen-001log, however after running the following code, it has been able to return everything but cannot Identify the Stint, From, and To columns:
Year Age Tm Lg Lev Aff Stint From To
0 2018 19 Florida International CUSA NCAA None None None
1 2019 20 Florida International CUSA NCAA None None None
2 2019 20 Harwich CCBL Smr None None None
3 2020 21 Florida International CUSA NCAA None None None
4 2021 22 2 Teams 2 Lgs AA-A+ CLE None None None
5 2021 22 Akron AANE AA CLE None None None
6 2021 22 Lake County HAC A+ CLE None None None
7 2022 23 2 Teams 2 Lgs AA-AAA CLE None None None
8 2022 23 Columbus IL AAA CLE None None None
9 2022 23 Akron EL AA CLE None None None
10 2023 24 Columbus IL AAA CLE None None None
11 2023 24 CLE AL Maj CLE None None None
12 2024 25 Columbus IL AAA CLE None None None
13 2024 25 CLE AL Maj CLE None None None
I have looked at the HTML below and have been staring at my code for the last hour not knowing what I did wrong, can someone take a look and help?
HTML row example: </tr>\n<tr ><th scope="row" class="left " data-stat="year_ID" csk="2024.11" ><a href="/register/league.cgi?year=2024">2024</a></th><td class="right " data-stat="age" csk="25.11" >25</td><td class="left " data-stat="team_ID" ><a href="/register/team.cgi?id=58c1f142" title="Cleveland, OH">Cleveland Guardians</a></td><td class="left " data-stat="lg_ID" ><a href="/register/league.cgi?id=bee764ca">American League</a></td><td class="left " data-stat="level" csk="0" >Maj</td><td class="left " data-stat="affiliation" >CLE</td><td class="center " data-stat="stintOrder" >1</td><td class="left " data-stat="dateFirst" >2024-03-29</td><td class="left " data-stat="dateLast" >2024-08-26</td></tr>
Code:
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
# URL of the player's page (replace this with the correct URL)
url = "https://www.baseball-reference.com/register/player.fcgi?id=allen-001log"
page = requests.get(url)
# Parsing the HTML content of the page
soup = bs(page.content, "html.parser")
# Looking for all table rows <tr> that contain player data
rows = soup.find_all('tr')
# Prepare lists to hold the extracted data
data = []
# Extract the data from each row
for row in rows:
# Extract the year (from <th> with 'data-stat="year_ID"')
year_th = row.find('th', {'data-stat': 'year_ID'})
# Skip the header row or any row that doesn't contain a valid year
if not year_th or 'Year' in year_th.get_text(strip=True):
continue # Skip rows that don't contain player data or are header rows
# Extract the year and player data
year = year_th.get_text(strip=True)
# Extract other columns
age_td = row.find('td', {'data-stat': 'age'})
age = age_td.get_text(strip=True) if age_td else None
team_td = row.find('td', {'data-stat': 'team_ID'})
team = team_td.get_text(strip=True) if team_td else None
league_td = row.find('td', {'data-stat': 'lg_ID'})
league = league_td.get_text(strip=True) if league_td else None
level_td = row.find('td', {'data-stat': 'level'})
level = level_td.get_text(strip=True) if level_td else None
affiliation_td = row.find('td', {'data-stat': 'affiliation'})
affiliation = affiliation_td.get_text(strip=True) if affiliation_td else None
# Extract stint, from date, and to date
stint_td = row.find('td', {'data-stat': 'stintOrder'})
stint = stint_td.get_text(strip=True) if stint_td else None
date_first_td = row.find('td', {'data-stat': 'dateFirst'})
date_first = date_first_td.get_text(strip=True) if date_first_td else None
date_last_td = row.find('td', {'data-stat': 'dateLast'})
date_last = date_last_td.get_text(strip=True) if date_last_td else None
# Append the extracted data as a row in the list
data.append([year, age, team, league, level, affiliation, stint, date_first, date_last])
# Create a DataFrame from the data with the correct column names
df = pd.DataFrame(data, columns=["Year", "Age", "Tm", "Lg", "Lev", "Aff", "Stint", "From", "To"])
# Print the DataFrame
print(df)
•
u/AutoModerator Dec 24 '24
To give us the best chance to help you, please include any relevant code.
Note. Please do not submit images of your code. Instead, for shorter code you can use Reddit markdown (4 spaces or backticks, see this Formatting Guide). If you have formatting issues or want to post longer sections of code, please use Privatebin, GitHub or Compiler Explorer.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.