Error: CourseCollectionsJson and CourseTopicUrlsList Urls are not equal about educative.io_scraper

[MoeNine]

"CourseCollectionsJson and CourseTopicUrlsList Urls are not equal"
Because some courses have assessment pages which this scraper may not support.

I have a simple solution, Just bypass the assessment pages. (i don`t need those assessment pages actually)

In src/ScraperType/CourseTopicScraper/CourseTopicScraperMain.py -> function "scrapeCourse"
Below the "topicUrlsList = self.apiUtils.getCourseTopicUrlsList(textFileUrl, courseUrl)"
Add those code

            for item in topicUrlsList:
                if 'assessment' in item:
                    topicUrlsList.remove(item)

then restart the scraper.

[buscon]

thank you for your answer and suggestion.
I edited that file as you suggested but now I get the following error:

$ python setup.py --run

            Educative Scraper (v3.2.5 Master Branch), developed by Anilabha Datta
            Project Link: https://github.com/anilabhadatta/educative.io_scraper/tree/v3-dev
            Check out ReadMe for more information about this project.
            Use the GUI to start scraping.

2023-11-24 16:44:24,446 - INFO - HomeScreen - Creating Home Screen...
2023-11-24 16:44:29,853 - INFO - StartScraper - StartScraper Initiated...
To Terminate, Click on Stop ScraperType Button

2023-11-24 16:44:29,942 - INFO - ExtensionScraper - ExtensionScraper initiated...
2023-11-24 16:44:29,942 - INFO - ExtensionScraper - Started Scraping from Text File URL: https://www.educative.io/courses/intermediate-javascript?showContent=true
2023-11-24 16:44:29,943 - INFO - BrowserUtility - Loading Browser...
2023-11-24 16:44:29,946 - ERROR - StartScraper - start: 20: ExtensionScraper:start: 49: BrowserUtility:loadBrowser: 48: Chromedriver might not be running in background, Please click on Start Chromedriver.
2023-11-24 16:44:35,255 - INFO - HomeScreen - Starting Chrome Driver...
Path: /home/marcello/Documents/Softwares/educative.io_scraper/src/ChromeDrivers/linux/chromedriver-linux64/chromedriver

Option “-e” is deprecated and might be removed in a later version of gnome-terminal.

Use “-- ” to terminate the options and put the command line to execute after it.

2023-11-24 16:44:37,384 - INFO - StartScraper - StartScraper Initiated...
To Terminate, Click on Stop ScraperType Button

2023-11-24 16:44:37,467 - INFO - ExtensionScraper - ExtensionScraper initiated...
2023-11-24 16:44:37,467 - INFO - ExtensionScraper - Started Scraping from Text File URL: https://www.educative.io/courses/intermediate-javascript?showContent=true
2023-11-24 16:44:37,467 - INFO - BrowserUtility - Loading Browser...
2023-11-24 16:44:38,087 - INFO - BrowserUtility - Browser Initiated
2023-11-24 16:44:41,712 - INFO - ApiUtility - Course Type Selector: a[href*='/courses/']
2023-11-24 16:44:51,876 - ERROR - StartScraper - start: 20: ExtensionScraper:start: 52: ExtensionScraper:scrapeCourse: 64: ApiUtility:getCourseUrl: 134: Message:
Stacktrace:
#0 0x5596f119be23
#1 0x5596f0ec47a7
#2 0x5596f0f031d3
#3 0x5596f0f032c1
#4 0x5596f0f3ea04
#5 0x5596f0f2403d
#6 0x5596f0f3c369
#7 0x5596f0f23de3
#8 0x5596f0ef7a7b
#9 0x5596f0ef881e
#10 0x5596f115d638
#11 0x5596f1161507
#12 0x5596f116bc4c
#13 0x5596f1162136
#14 0x5596f11309cf
#15 0x5596f1185b98
#16 0x5596f1185d68
#17 0x5596f1194cb3
#18 0x7f4dcf494ac3

[buscon]

I tried with another course and it works.
the above error seems related to that course.

[MoeNine]

Do Not Use:https://www.educative.io/courses/intermediate-javascript
Use https://www.educative.io/courses/intermediate-javascript/getting-started

Use the course`s first topic url but not course url.

[buscon]

ok, thank you!

[buscon]

I actually have a similar problem with this course: in this case it does not work even if I set the first page of the course as a download link.
https://www.educative.io/courses/fundamentals-of-digital-signal-processing
or
https://www.educative.io/courses/fundamentals-of-digital-signal-processing/the-world-of-signals

[anilabhadatta]

@buscon checking

[anilabhadatta]

@buscon fixed in latest commit, do git pull

[buscon]

I just tested and yes, I confirm it works. thank you!