leagify / prospect-scraper-dt2021 Goto Github PK
View Code? Open in Web Editor NEWScraping draft prospects for fun and charts.
Scraping draft prospects for fun and charts.
Sometimes, the site gets updated, but the date on the page does not.
Create a second command line parameter to let the program ignore the parsed date and use today instead.
Since command line parameters are currently optional, this parameter should take that into account.
I assume we should name this variable something like ignorePageDate.
The RunTheBigBoards() and RunTheMockDraft() should take this variable as a parameter.
Applicable code is here:
prospect-scraper-dt2021/src/prospectScraper/Program.cs
Lines 16 to 52 in 5c5b582
The README.md has this project as .NET Core 3.1
"This program is written in .NET Core 3.1."
While prospectScraper.csproj has the framework as .NET 5
'TargetFramework>net5.0</TargetFramework'
https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/switch-expression
This starts a pretty gnarly switch statement:
Change this to an expression.
A testing project exists, but there's only one test that returns true no matter what. Add at least one .NET unit test.
When building, there's a warning message:
Warning: /home/runner/work/prospect-scraper-dt2021/prospect-scraper-dt2021/src/prospectScraper/MockDraftPick.cs(149,19): warning CS8632: The annotation for nullable reference types should only be used in code within a '#nullable' annotations context. [/home/runner/work/prospect-scraper-dt2021/prospect-scraper-dt2021/src/prospectScraper/ProspectScraper.csproj]
It's annoying, and I'd like to fix that so it doesn't complain any more.
I recently opened the project in GitPod and some of the stuff failed. It would be nice to check if the project builds in Linux and can run tests in Linux.
This year, prospect rankings and mock draft picks should have information about whether or not their schools have decided to participate in games this year.
This information should match the property names that are set up in #14 .
For example, ProspectRanking looks like this:
prospect-scraper-dt2021/src/prospectScraper/ProspectRanking.cs
Lines 6 to 18 in 0d4a7d5
additional properties will likely be:
Likely, the ProspectMap will also need to have the additional properties.
Here's how it looks so far:
prospect-scraper-dt2021/src/prospectScraper/ProspectRankingMap.cs
Lines 8 to 29 in 0d4a7d5
Data regarding NCAA conference participation in football games is represented in CovidOptOut.cs.
Using CSVHelper, create a way to get this data into the program. In this case, we only need to read the information in. We do not need to write this information back out to a file.
Probable field names:
There are examples of implementations for this, for example, in School.cs:
prospect-scraper-dt2021/src/prospectScraper/School.cs
Lines 1 to 29 in 0d4a7d5
https://devblogs.microsoft.com/dotnet/announcing-net-5-0-rc-1/
Currently, I think .NET Core is being installed in the Dockerfile with the normal workspace.
The .NET 5 RC will probably need to be installed separately.
prospectScraper.csproj in src/ and prospectScraperTest.csproj in test/ will need to be updated in order for this to work correctly.
This file was originally created when entire football conferences started cancelling or postponing their seasons. Some of those conferences have now backtracked on their decisions, so it might be better to change this CSV file to reflect that.
Original file is here:
https://github.com/Leagify/prospect-scraper-dt2021/blob/master/src/prospectScraper/info/CovidOptOuts.csv
Current Header is:
Conference,FallFootball,SpringFootball
A more accurate header would probably be:
School,Conference,2020ScheduledGames
Using SchoolStatesAndConferences.csv would be a good start:
https://github.com/Leagify/prospect-scraper-dt2021/blob/master/src/prospectScraper/info/SchoolStatesAndConferences.csv
You wouldn't need to do the state.
Also, D2 and D3 schools are not necessary. The only exception would be schools that have prospects actually appear in either the mock draft or big board, and those players tend to be few and far between. The argument could also be made for FCS schools to be omitted until a player is mentioned, but I know that Trey Lance from NDSU is already a hotly discussed prospect.
Oops.
Add an extra div to the third round.
Unhandled exception. System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.GetResponse()
at HtmlAgilityPack.HtmlWeb.Get(Uri uri, String method, String path, HtmlDocument doc, IWebProxy proxy, ICredentials creds)
at HtmlAgilityPack.HtmlWeb.LoadUrl(Uri uri, String method, WebProxy proxy, NetworkCredential creds)
at HtmlAgilityPack.HtmlWeb.Load(Uri uri, String method)
at HtmlAgilityPack.HtmlWeb.Load(String url, String method)
at HtmlAgilityPack.HtmlWeb.Load(String url)
at prospectScraper.ProspectScraper.RunTheMockDraft(Boolean parseDate) in /workspace/prospect-scraper-dt2021/src/prospectScraper/ProspectScraper.cs:line 89
at prospectScraper.Program.Main(String[] args) in /workspace/prospect-scraper-dt2021/src/prospectScraper/Program.cs:line 42
Attempted to run a scrape for a new mock draft, but both the big board and mock draft had this issue tonight. I hope it gets resolved magically, but if not, I'll need to figure out why stuff would magically start timing out when the web page loads fine in a browser.
In the info directory add a new csv file to track whether the College is going to be playing football in the fall.
Conferences that aren't playing in the fall (AFAIK):
It seems like doing this by conference will be a good way to go, as the conference is listed in SchoolStatesAndConferences.csv - I'm not 100% sure that information is in the prospect object, though, so there may need to be some work in integrating all this together afterward.
First things first.
Create the CSV with two fields:
Conference,FallFootball
string, bool
If schools start trying to have seasons in the spring or doing something else, I'm assuming that I'll just add another column to track that as a boolean as well.
Once this has been created, I'll need to make some additional issues to integrate this into the actual process of scraping, as (at this moment) this is information that is not being scraped from the site, but added to the obtained data afterwards.
https://devblogs.microsoft.com/dotnet/announcing-net-5-0/
I think we're currently on the 5 RC, make any updates necessary to get onto the LTS version.
Any reason the prospectScraperTest.csproj (test project) is not included in the prospectScraper.sln (main solution)?
I was making a change on another issue and noticed this, if the test project was part of the main solution, any contributors using Visual Studio to develop would see the test project regardless of what they're making a change to and would prompt contributors to be more likely to run the test before commiting changes (I would hope ๐).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.